Для чего нужен datapower ibm

Для чего нужен datapower ibm

An industry leading, high security application gateway for modern, traditional and hybrid cloud workloads.

See it in action (03:16) Read the data sheet

What is IBM DataPower Gateway?

With web application attacks making up 39% of all breaches in 2021¹, IBM® DataPower® Gateway helps businesses meet their security and integration needs. Enterprise-grade security, control and comprehensive transport and data protocol support include IBM MQ, Apache Kafka and traditional business critical data formats.

Businesses across the world depend on IBM DataPower Gateway 400+

financial institutions worldwide use IBM DataPower.

different industries have companies that use IBM DataPower.

countries with DataPower deployments.

Benefits Mitigate vulnerabilities

Enterprise-grade security with the highest level of assurance certification to protect your critical business applications.

Accelerate transformation

High-speed message transformation and transport protocol bridging helps deploy news services rapidly.

Reduce cost and complexity

A single, drop-in gateway helps simplify the topology and operations, resulting in cost savings and reduced risk.

Simplify troubleshooting

Speed problem determination with near-real time visibility of transactions through centralized operations.

Features Runtime application security

Enforce security across messaging protocols with a DMZ-ready gateway.

Transform messages and protocols

Enables message and protocol mediation by bridging modern and traditional application standards.

Application and service protection

Ensure consistent application performance and service protection with traffic control and routing.

Access management and control

Authentication and authorization decision point for all standards, including OAuth and LDAP.

Extensive Debug Tool

Allows customers to inspect DataPower transactions in full detail including payload and all metadata information

Intuitive user experience

A flexible, configuration-driven experience that reduces the skills barrier and speeds time to value.

Use Cases

Secure messages Comply with the latest industry security standards for the highest level of protection against attacks.

Control traffic Through simple configuration options, IBM DataPower provides message level throttling to back-end applications, ensuring no system is overrun by inbound messages, preventing unplanned outages.

Integrate workloads Comprehensive transport and data protocol support including IBM MQ and Apache Kafka as well as traditional, business critical data formats.

Optimize applications With application caching, data can be stored in the gateway, reducing the amount of message traffic flowing to applications, improving response times while reducing overall network bandwidth.

Product versions

IBM DataPower Gateway versions

Physical

Hardware offering for the DMZ with specialized security and message optimization.

Virtual

Physical appliance capabilities as containers or virtual machines across any deployment.

Physical

• Intrusion detection: hardware protection against malicious altering of physical system
• Secured boot process: Trusted Platform Module (TPM) chip decrypts firmware at startup
• Hardware Security Module (HSM): provides tamper-proof storage of private key material
• FIPS 140-2 Level 3 compliance through the use of HSM
• Purpose-built system providing hardware-accelerated operations
• Lower latency and higher throughput than virtual appliances

Virtual

• Deployed on commodity x86 hardware servers and supported cloud environments
• High elasticity—easily moved between servers with flexibility to increase capacity
• Flexible licensing and entitlement options
• Increased workload isolation
• Run multiple instances concurrently on a single physical server
• Provides lower-cost environment for application development and test validation

Case studies Federal Bank

Federal Bank creates an API banking system to better integrate with other organizations and ecosystems

1LINK enables fast, affordable integration with API platform.
Q8 Petroleum Italia
Q8 Italia leads an enterprise-wide move to IBM Cloud Pak for Integration.
Resources DataPower Highlights
Get a one-page summary of the key features and benefits of IBM DataPower Gateway.
Download the Infographic Learn more
Learn more about why IBM DataPower is the industry leading application gateway.
Read the DataSheet Get started
Evaluate IBM DataPower Gateway with a free container available in the IBM Entitled Registry.
Related products IBM API Connect®
Get this API management solution for your entire API lifecycle from creation to management.
IBM App Connect
Connect all of your apps and data with this simple, smart and scalable integration solution.
IBM Event Automation

Enable business and IT teams to put events to work, detecting and responding to new trends, customer issues or competitive threats in real-time.

Для чего нужен datapower ibm

An industry leading, high security application gateway for modern, traditional and hybrid cloud workloads.

See it in action (03:16) Read the data sheet

What is IBM DataPower Gateway?

With web application attacks making up 39% of all breaches in 2021¹, IBM® DataPower® Gateway helps businesses meet their security and integration needs. Enterprise-grade security, control and comprehensive transport and data protocol support include IBM MQ, Apache Kafka and traditional business critical data formats.

Businesses across the world depend on IBM DataPower Gateway 400+

financial institutions worldwide use IBM DataPower.

different industries have companies that use IBM DataPower.

countries with DataPower deployments.

Benefits Mitigate vulnerabilities

Enterprise-grade security with the highest level of assurance certification to protect your critical business applications.

Accelerate transformation

High-speed message transformation and transport protocol bridging helps deploy news services rapidly.

Reduce cost and complexity

A single, drop-in gateway helps simplify the topology and operations, resulting in cost savings and reduced risk.

Simplify troubleshooting

Speed problem determination with near-real time visibility of transactions through centralized operations.

Features Runtime application security

Enforce security across messaging protocols with a DMZ-ready gateway.

Transform messages and protocols

Enables message and protocol mediation by bridging modern and traditional application standards.

Application and service protection

Ensure consistent application performance and service protection with traffic control and routing.

Access management and control

Authentication and authorization decision point for all standards, including OAuth and LDAP.

Extensive Debug Tool

Allows customers to inspect DataPower transactions in full detail including payload and all metadata information

Intuitive user experience

A flexible, configuration-driven experience that reduces the skills barrier and speeds time to value.

Use Cases

Secure messages Comply with the latest industry security standards for the highest level of protection against attacks.

Control traffic Through simple configuration options, IBM DataPower provides message level throttling to back-end applications, ensuring no system is overrun by inbound messages, preventing unplanned outages.

Integrate workloads Comprehensive transport and data protocol support including IBM MQ and Apache Kafka as well as traditional, business critical data formats.

Optimize applications With application caching, data can be stored in the gateway, reducing the amount of message traffic flowing to applications, improving response times while reducing overall network bandwidth.

Product versions

IBM DataPower Gateway versions

Physical

Hardware offering for the DMZ with specialized security and message optimization.

Virtual

Physical appliance capabilities as containers or virtual machines across any deployment.

Physical

• Intrusion detection: hardware protection against malicious altering of physical system
• Secured boot process: Trusted Platform Module (TPM) chip decrypts firmware at startup
• Hardware Security Module (HSM): provides tamper-proof storage of private key material
• FIPS 140-2 Level 3 compliance through the use of HSM
• Purpose-built system providing hardware-accelerated operations
• Lower latency and higher throughput than virtual appliances

Virtual

• Deployed on commodity x86 hardware servers and supported cloud environments
• High elasticity—easily moved between servers with flexibility to increase capacity
• Flexible licensing and entitlement options
• Increased workload isolation
• Run multiple instances concurrently on a single physical server
• Provides lower-cost environment for application development and test validation

Case studies Federal Bank

Federal Bank creates an API banking system to better integrate with other organizations and ecosystems

1LINK enables fast, affordable integration with API platform.
Q8 Petroleum Italia
Q8 Italia leads an enterprise-wide move to IBM Cloud Pak for Integration.
Resources DataPower Highlights
Get a one-page summary of the key features and benefits of IBM DataPower Gateway.
Download the Infographic Learn more
Learn more about why IBM DataPower is the industry leading application gateway.
Read the DataSheet Get started
Evaluate IBM DataPower Gateway with a free container available in the IBM Entitled Registry.
Related products IBM API Connect®
Get this API management solution for your entire API lifecycle from creation to management.
IBM App Connect
Connect all of your apps and data with this simple, smart and scalable integration solution.
IBM Event Automation

Enable business and IT teams to put events to work, detecting and responding to new trends, customer issues or competitive threats in real-time.

Обучающий курс по DataPower

В 2017 году, когда начинался наш проект во Вьетнаме, мы столкнулись с новым для нас зверем IBM DataPower. IBM DataPower – продукт, представляющий собой gateway между клиентами и бэкендами, предназначенный для фильтрации, маршрутизации, обогащения или других преобразований проходящих через него сообщений (далее – запросов). Обучаться нужно было быстро, времени на раскачку не было, поэтому нам было предложено самостоятельно ознакомиться с ним, после чего были многочасовые конференции по скайпу с нашим коллегой из Москвы, который передавал нам свои знания и опыт работы с этим продуктом.

Самостоятельное обучение основывалось на изучении документации и просмотре обучающих видео из интернета – и тут меня ждал подвох. Мне практически не удалось найти информации на русском языке. К слову, мои знания английского языка на тот момент были не на высшем уровне, к тому же это был мой первый проект и, наверное, именно эти факторы усложнили мне жизнь. Это сподвигло меня написать обучающую статью на русском языке и в максимально простом изложении для начинающих разработчиков, которые столкнулись с этим продуктом и пытаются оперативно понять его азы. Статья не освободит вас от чтения документации, но облегчит жизнь на первых этапах понимания «как это работает».

Стоит также заметить, что приведенная в практике структура будет приближена к реальному проекту, что позволит вам использовать ее как базу, расширяя и дополняя под ваши требования. В заключение к разделам «Теория» будет приведено несколько слов об уже реализованном проекте, а также некоторые особенности, на которые стоит обратить внимание.

Часть 1. Установка DataPower средствами Docker Toolbox

Установите и запустите приложение Docker Toolbox. Сразу после запуска вы увидите IP адрес машины, по которому в дальнейшем будет доступен DataPower:

Для запуска образа необходимо изменить некоторые настройки виртуальной машины (актуально для версии IDG.2018.4.1.0) и перезапустить ее:

Останавливаем докер-машину командой:

docker-machine stop

docker-machine start

Примечание. Если будет предложено перезапустить «docker-machine env» — выполните:

docker-machine env

docker pull ibmcom/datapower

docker run -it \ -v $PWD/config:/drouter/config \ -v $PWD/local:/drouter/local \ -e DATAPOWER_ACCEPT_LICENSE=true \ -e DATAPOWER_INTERACTIVE=true \ -p 9090:9090 \ -p 3630:3630 \ -p 9022:22 \ -p 7170:7170 \ --name idg \ ibmcom/datapower 

web-mgmt 0 9090

Часть 2. Домены

2.1. Теория

Домены в DataPower позволяют разделить средства администрирования и разработки, а также обеспечить безопасность.

После установки существует только домен default, из которого могут быть созданы домены приложений. Для каждого домена существует своя конфигурация параметров.

Некоторые общие ресурсы и параметры можно определить только в default домене, к ним относятся сетевые интерфейсы, пользователи и управление доступом, домены приложений и другие.

Домен приложений – это раздел разработки для служб, обрабатывающих запросы. Службы, определенные в нём, не могу быть общими с другим доменом приложений. Домены приложений могут перезапускаться отдельно и независимо друг от друга, без необходимости полного перезапуска DataPower.

Вы можете создавать, перезапускать, сбрасывать, приостанавливать, возобновлять или удалять домены. Более подробную информацию обо всех возможностях администрирования вы можете найти в официальной документации.

Немного о реализованном проекте. Мы использовали 3 домена:

• default – домен по умолчанию, содержащий общие ресурсы и параметры;
• trunk – основной домен, содержащий все необходимое для обработки запросов;
• settings – домен настроек и безопасности, в локальных файлах содержится информация о правилах маршрутизации сервисов и параметрах безопасности.

Необходимость перенести все настройки в отдельный домен возникла в связи с поиском более простого пути развертывания. Как и во многих проектах среды dev, test и prod у нас были разделены, а вынос в отдельный домен настроек позволил устанавливать все основные домены из dev среды в других средах посредством экспорта/импорта, без риска потерять настройки среды.

2.2. Практика

• В поле поиска введите «domain», выберите «Application Domain» и нажмите «Добавить»
  
• Здесь вам необходимо указать название домена, комментарий (при желании) и активировать auditing и logging. Заполните поля и примените изменения
  
• Аналогично добавьте еще один домен «trunk»
• Перейдите в Review changes
  
• и сохраните конфигурацию домена
  
• Проверить состояние созданных объектов вы можете, перейдя в default домене по дереву навигации в Status -> Main -> Object Status
  
• Выберите View by: types
  
• Найдите в списке Application Domain и проверьте состояние объектов: каждый из них должен быть сохранен, включен и в состоянии up. Если всё так – переходите к следующей главе

Часть 3. Менеджеры очередей

Менеджер очередей не является обязательным компонентом IBM DataPower, но именно на примере с MQ можно показать всю мощь этого продукта. MQ будем использовать от компании IBM. Во время тестирования, описанного в 6 главе, нам будет необходимо отправлять сообщение в локальный менеджер очередей. В данной статье я сделаю это с помощью утилиты rfhutil, но вы можете использовать любой доступный вам способ. Для тестирования необходимо будет создать коннект из DP к вашему локальному менеджеру очередей с помощью MQ Queue Manager.

3.1. Теория

Менеджер очередей обеспечивает обмен данными между шлюзом и удаленными администраторами очередей.

Вы также можете настроить MQ Queue Manager Group, что позволит повысить отказоустойчивость системы. Это может быть полезно, например, если вы хотите подключать клиента к любому из набора работающих администраторов очередей и еще в ряде случаев, с которыми вы можете ознакомиться в официальной документации.

Из опыта проекта стоит отметить только одну особенность: в свое время мы хотели попробовать реализовать балансировку нагрузки средствами DataPower, в частности, с использованием групп менеджеров очередей, но на практике не обнаружили такой возможности. Альтернативное решение – создание кластера менеджеров очередей.

3.2. Практика

3.2.1. Подготовка

1. Установите WebSphere MQ;
2. Создайте локальный менеджер очередей LOCAL_DP_QM, доступный по порту 3630;
3. Настройте канал DP.SVRCONN;

При создании канала вам могут быть полезны следующие команды:

strmqm LOCAL_DP_QM /*запуск менеджера mq*/ runmqsc LOCAL_DP_QM /*вход в оболочку mq*/ DEFINE CHANNEL (DP.SVRCONN) CHLTYPE(SVRCONN) MCAUSER(‘evlasenko’) /*создание канала*/ SET CHLAUTH(‘DP.SVRCONN’) TYPE(USERMAP) CLNTUSER(‘evlasenko’) USERSRC(channel) ADDRESS(‘*’) ACTION(ADD) /*включение доступа для пользователя со стороны клиента*/ SET CHLAUTH(DP.SVRCONN) TYPE(BLOCKUSER) USERLIST(‘nobody’) /*отключить список блокировки пользователей*/ ALTER LISTENER (SYSTEM.DEFAULT.LISTENER.TCP) TRPTYPE(TCP) PORT(3630) control(QMGR) /*включение автозапуска листенера порт 3630*/ ALTER AUTHINFO (SYSTEM.DEFAULT.AUTHINFO.IDPWOS) AUTHTYPE(IDPWOS) CHCKCLNT(OPTIONAL) /*выключить аутентификацию*/ REFRESH SECURITY TYPE(CONNAUTH) /*обновить настрйоки безопасности*/ endmqm LOCAL_DP_QM /*остановка менеджера mq*/ strmqm LOCAL_DP_QM /*запуск менеджера mq*/ END 

DP.SVRCONN/TCT/127.0.0.1(3630)

3.2.2. Создание IBM MQ Queue Manager

1. Перейдите в домен trunk.
2. В поиске наберите MQ, выберите IBM MQ Queue Manager и нажмите добавить.
3. Вам необходимо указать название(TEST_QM), хостнейм менеджера очередей, имя менеджера очередей и имя канала, а также таймаут. Выполните настройку и сохраните изменения.

Проверьте состояние объекта менеджера очередей аналогично проверке статуса доменов. Для этого из домена trunk выберите Object Status и фильтр View by: types. В разделе «IBM MQ Queue Manager» отыщите соответствующий объект и проверьте его состояние.

Часть 4. Многопротокольные шлюзы

4.1. Теория

Multi-Protocol Gateway (MPG) – многопротокольный шлюз, позволяющий принимать запросы от клиентов по различным протоколам, а затем по различным протоколам передавать их на удаленный сервер. Протокол, используемый клиентом может не совпадать с протоколом, используемым сервером удаленного доступа.

В основных настройках MPG вы можете определить следующие компоненты:

• XML Manager – управляет компиляцией и кэшированием таблиц стилей(xsl,xslt), кэшированием документов.
• Policy – состоит из правил, каждое из которых определяет набор действий, применяемых к сообщению, проходящему через шлюз.
• Настройки front и back side (настройка url, типов входящего и исходящего сообщений, таймаутов и другого).

Пара слов о проекте:

В реализации проекта используется 4 многопротокольных шлюза (маршрутизирующий, 2 трансформационных для разных конечных систем и дополнительный, предназначенный для получения файлов из домена settings). На схеме ниже представлена общая схема взаимодействия:

Количество MPG может варьироваться в зависимости от архитектуры решения в целом. В нашем случае DataPower стоит перед интеграционной шиной (IIB) и микросервисами, которые имеют существенные различия в интерфейсах(json/http против xml/mq), поэтому трансформационные MPG было решено делать под каждый специфический бэкенд и называть его соответствующе. Для всех клиентов мы работаем по json/http, поэтому маршрутизирующий MPG – один. Основные MPG состоят из 3 правил обработки сообщений – запроса, ответа и ошибок. Каждое правило состоит из необходимых действий, таких как transformation, logging, routing и прочие.

Из особенностей – если в policy вы используете действие ConvertQueryParamToXML, то будьте внимательны к InputConversion. Если вы зададите Default Encoding значение JSON и попытаетесь отправить GET-запрос, то будете удивлены, обнаружив, что сообщение не претерпело никаких трансформаций, указанных вами и не обнаружите никаких его следов. Данную особенность поможет преодолеть создание отдельного правила для GET-запросов.

4.2. Практика

4.2.1. Подготовка

Все необходимые для работы файлы вы можете найти по ссылке https://github.com/EvgenyaVlasenko/IBM_DataPower.git

4.2.1.1. Домен trunk

1. Перейдите в домен trunk.
2. На панели управления выберите File Management.
3. В каталоге local создайте следующую структуру каталогов и поместите в нее соответствующие файлы (в каждом файле содержится краткое описание того, какие функции он выполняет, а также более подробно остановимся на этом далее в статье).

4.2.1.2. Домен settings

1. Перейдите в домен settings.
2. На панели управления выберите File Management.
3. В каталоге local создайте следующую структуру каталогов и поместите в нее соответствующие файлы (внутри файлов также содержится их краткое описание).

4.2.2. Создание GetFileMPG

Для начала создадим простой вспомогательный MPG, который будет возвращать файлы из домена settings.

1. Перейдите в домен settings.
2. На панели управления выберите Multi-Protocol Gateway и нажмите создать.
3. Укажите имя (GetFileMPG), описание(по желанию) и тип бэкенда (dynamic). На самом деле, так как вызова бэкенда фактически не будет, а будет только возвращаться файл из локальной системы, то в данном примере тип бэкенда можно указать любой.
4. Задайте типы запроса и ответа. Явное указание типов позволит сократить количество встроенных проверок. Указание типа Pass through позволит не создавать правило (в данном случае для трансформации ответа). В случае, если мы укажем Request Type тоже Pass through, то не сможем никак обработать сообщение. Этот вариант нам не подходит, поэтому ограничиваем тип запроса с помощью Non-XML.

Нажмите на + и выберите HTTP Handler для создания нового Front Side Protocol.

Здесь вам следует указать имя, ip-адрес, порт и список разрешенных методов. Следует обратить внимание на ip-адрес. Если указать 0.0.0.0, то обращаться к этому шлюзу могут все. Если 127.0.0.1 – только другие шлюзы внутри этого же DataPower. Так как среди настроек есть параметры безопасности, то используем именно второй вариант. Заполните поля и нажмите «Применить», протокол автоматически добавится в шлюз.

Нажмите на + для добавления новой политики.

curl -vv -k "http://192.168.99.100:7170/trunk/route/routeRules.xml" 

4.2.3. Создание RoutingMPG

Теперь создадим RoutingMPG. На основании входного запроса и правил маршрутизации он определит, куда и с какими параметрами следует отправить запрос.

1. Перейдите в домен trunk.
2. Создайте новый Multi-Protocol Gateway в соответствии с пунктами 2-10 раздела 4.2.2, используя следующие значения:
   • 3 – имя: RoutingMPG, тип бэкенда: Dynamic (для возможности маршрутизации запросов в разные MPG при необходимости).
   • 4 – Rq: Non-xml, Rs: Non-xml.
   • 6 – имя: RoutingHTTP_FSH, ip: 0.0.0.0, порт: 7170, + Get method.
   • 8 – имя: RoutingPolicy.
   • 9 – имя: RoutingPolicy_rule_req, направление: Client to Server.

3. Добавьте еще одно действие для маршрутизации запроса, для этого перетащите на правило действие «Route», дважды щелкните на нем для настройки, заполните поля и примените изменения. Файл route.xsl получает файл настроек роутинга из домена settings через созданный ранее GetFileMPG. После этого на основании URI выделяются из файла уже те настройки, которые необходимы для данной операции. Часть из них используется для маршрутизации, а часть добавляется в хедеры для использования в других MPG. Параметры input и output определяют способ работы только с телом сообщения и никак не влияют на хедеры и переменные. Поэтому: вход из null – так как для роутинга не используется информация из тела сообщения. Вывод в null – так как результатом трансформации является только изменение служебной информации.

curl -vv -k "http://192.168.99.100:7170/dp/test/transformMessage"

Окончательный результат политики должен выглядеть следующим образом:

Часть 5. Тестирование

5.1. Подготовка

1. Скачайте, например, утилиту rfhutil для чтения и записи сообщений в очередь;
2. Тестовые файлы находятся в папке tests той же директории, что и файлы проекта.

5.2. Проверка работоспособности

Отправьте запрос помощью утилиты curl (для запроса ниже текущая директория должна быть той же, где лежит example.json).

curl -vv -k "http://192.168.99.100:7170/dp/test/transformMessage" -H "Content-Type: application/json" --data-binary @example.json

Если ничего не помогло – задавайте вопросы в комментариях.

Monitoring IBM Datapower Gateways

This article describes the fundamentals of monitoring the health and capacity of IBM Datapower Gateways, including why to monitor and best practices for monitoring.

Content

Introduction

IBM Datapower Gateways (hereafter called DataPower) is a purpose-built hardware platform designed to simplify, secure, and accelerate XML, Web services, and Enterprise Service Bus deployments.

As with other network appliances, monitoring the health and capacity of DataPower appliances will ensure that they are ready to perform the functions for which they are configured. Monitoring not only notifies administrators of exceptions, it also provides trending analysis for managing the appliances and their capacity utilization over time, thus enabling the organization to maximize its return-on-investment and receive warnings of increases in network volumes and potential capacity issues.

This article describes various DataPower status inquiry methods and presents strategies and best practices for interpreting them. This article is based on DataPower Firmware Revision 3.8.0. Monitoring status providers may change with enhancements to the firmware, so you should check current firmware documentation for any additions to monitoring components.

Why monitor?

The DataPower Appliance family consists of 1U rack-mountable network devices. The latest generation devices (9235/9004 class) contain four gigabit RJ-45 Ethernet interfaces, a DB-9 Serial port, hot swappable power supplies and fan-trays, batteries, eight gigabytes of RAM, compact flash-based file system, and other components within a tamper-proof case. Optional features including internal hard drives, hardened cryptographic modules, and additional compact flash bays.

Each of these components helps ensure that the device is properly configured for the amount of network data it receives. Knowing that the devices are functioning properly ensures that they are available and ready to process this traffic. For example, if you are alerted to variations in the performance of the device’s fans, you may avoid having to take the device offline for unanticipated service. Understanding the level of network traffic and being aware of incremental changes may avoid bottlenecks as traffic increases over time.

Monitoring fundamentals on DataPower

DataPower provides a variety of information regarding general system health as well as consumption of resources and services. Physical parameters range from the temperature of CPUs, utilization of memory and file system, interface utilization, and voltage reading, among other physical values. In addition, there are more formulaic indicators, such as System Usage, which is a calculation of system capacity.

DataPower exposes these status values in a variety of ways. You can use the Web GUI or Command Line Interface (CLI) show commands to browse a list of status values. Or you can use the XML Management Interface (XMI) to send SOAP messages containing dp:get-status requests to the device, which responds with status information contained in SOAP responses. DataPower also supports the Simple Network Management Protocol (SNMP) and acts as an SNMP agent, providing status information in response to SNMP operations and in the creation of alerts via the SNMP notification mechanism.

Figure 1 shows the CPU usage as displayed within the Web GUI. It is obtained by navigating from Status Menu => System => CPU Usage. The data is displayed in a table incrementing from the latest 10 seconds through to the latest 24 hours.

Figure 1. Web GUI CPU usage display

The CLI show commands are used to display status information, and Listing 1 shows the show cpu command, which provides the same table of data shown in the Web GUI:

Listing 1. CLI Show CPU command

xi50# show cpu 10 sec 1 min 10 min 1 hour 1 day cpu usage (%): 1 1 7 7 7

While the Web GUI and CLI are convenient tools to fetch status information interactively, the XMI can be programmatically integrated into more complex solutions. For example, a Java class could execute a dp:get-status request and perhaps perform configuration modification based on the response. The SOAP request in Listing 2 shows a dp:get-status request to fetch CPU usage status:

Listing 2 Sample get status XMI request

The response is returned in a SOAP payload, as shown in Listing 3 below. Again, the CPU status is returned within a subtree containing the same table of data returned by the Web GUI and CLI:

Listing 3 XMI dp:get-status response

    2009-09-24T11:56:22-04:00  1 1 1 1 1     

You can get a vast amount of status data using the dp:get-status request. For more information, including details of the schemas and WSDL used to customize dp:get-status and other XMI operations, see the IBM Redpaper WebSphere DataPower SOA Appliances: The XML Management Interface.

Most organizations query the health and capacity of a network device using the SNMP protocol in conjunction with tools such as those in the IBM Tivoli Monitoring (ITM) and Tivoli Composite Application Manager (ITCAM) product families. These tools use SNMP over UDP to poll an SNMP agent for device and application metrics. The management software may also receive notification alerts from the agent in response to particular events happening on the device. The DataPower appliance may be configured to act as a SNMP agent, responding to inbound polling requests and sending alerts in response to preconfigured events.

SNMP status variables are organized in hierarchies, which are described by the Management Information Base (MIB) document. Each metric that can be polled is addressed by an Object Identifier (OID). Some metrics are scalar objects describing a single data point, such as the current firmware version on the appliance. Other metrics may be tabular, such as the CPU status provided in the previous examples. When a specific OID is known, a GET OID can be used by the SNMP manager to get the specific metric. If all metrics in a specific hierarchy are desired, a Get Subtree can be used to get all values within that hierarchy. The DataPower appliance provides three Enterprise MIB documents for configuration, status, and notification. It is the status MIB that we are interested in.

While status inquiry is a straightforward endeavor, alerting is done using several DataPower objects. The device has four built-in notification alerts: authenticationFailure, linkDown, coldStart, and linkUp. Others are preconfigured, as described below. A properly configured SNMP monitor receives these traps in the event that the device restarts, its interfaces become enabled or disabled, or when a failed attempt to access the device occurs. In addition to the built-in alerts, custom alerts may be generated by subscribing to a list of error conditions or in conjunction with the logging system.

Reliance on alerts alone is not a sufficient monitoring strategy. For example, if the event causing the alert affects the device’s ability to send the message over the network, the notification may not be received at the SNMP monitor. Therefore it is prudent to combine subscription to alert messages with polling of status information, to provide a robust mechanism for communicating with monitoring tools.

Many status providers (or monitoring agents) are built into the DataPower firmware to fetch status data. Many providers are specific to the device. These providers (such as the environmental components, fans, temperatures, or battery health), are available within the default domain and are always enabled. Other status data (such as transaction rates for DataPower services) are segmented by application domain and may be further segmented by XML-Manager or DataPower service.

While the device-level data is automatically enabled, transaction data such as transaction rates or transaction times is usually available only when Statistics are enabled on the device. There are exceptions to this generalization — for example, CPU status requires statistic enablement, while System Load does not. Each domain must have its individual Statistics setting enabled to provide domain-specific status.

This section shows you how to enable monitoring of DataPower from SNMP tools, and how to produce SNMP alerts from within DataPower. You’ll see how Logging Target configuration can be configured to produce alerts based on system events, and how to subscribe to events such as Out of Memory or Power Supply failure to generate alerts. An example of a Power Supply failure will be used to demonstrate these principles.

SNMP settings must first be configured on the DataPower appliance. This configuration is accessible from the default domain and accessed from the left navigation menu of the DataPower Web GUI by first selecting the Administration menu and then selecting SNMP Settings under the Access heading.

This configuration consists of multiple tabs. The main tab must have the Admin State set to Enabled. Typically, the Local IP Address is set to a Host Alias defined in the default domain that maps to the Management Interface IP, which restricts SNMP polling requests to this IP and not any of the client traffic interfaces (eth0, eth1, or eth2). Figure 2 shows SNMP settings enabled on the default Local Port of 161. Outbound polling responses and traps will be sent out using any appliance interface that has the correct routing. To restrict this outbound traffic to the same IP, add a static route to the appliance’s mgt0 configuration.

Figure 2. Enabling SNMP settings

The DataPower MIBs can be downloaded from the appliance to be used by any SNMP management tool. The MIBs enable these tools to translate named objects such as dpStatusMemoryStatusUsage to an OID used to request the metric. All appliance status OIDs are in the drStatusMIB.txt MIB file. Figure 3 shows the Enterprise MIBs tab of the SNMP Settings screen, and the method for downloading the MIBs:

Figure 3. SNMP MIB download tab

The Trap Event Subscription tab contains a list of event codes that can be sent to the management software as an alert. Examples are the codes for «Internal cooling fan has stopped» or «Power supply failure.» Figure 4 below shows some of the default preloaded subscriptions. To add additional events, click Select Code. If a specific code is not shown in the list, you can add it manually. For example, adding code 0x806000e2 adds certificate monitor events to indicate when a certificate is nearing expiration. You can get these event codes from their associated log records in the default log. You can also get the event code in the Message Reference document for your firmware release.

Figure 4. SNMP Trap Event Subscription

The SNMPV1/V2c Communities tab defines access policies for management software using SNMP V1 and V2. The community name is used as a credential to access the SNMP data on the appliance. A common community name for read-only access is public. A DataPower domain, either the default or an application domain, can be associated with the configured community.

If application data is to be polled, specify the application domain; otherwise use the default domain. Specifying an application domain does not prevent management software from polling device-level metrics such as device load, CPU utilization, memory metrics, and environmental statistics. Additionally, it allows polling of application metrics such as transaction rates and times, MQ queue manager status, message counters, or SLM metrics.

The mode of the community should be configured as read-only for access to appliance status metrics. Finally, a remote host access of 0.0.0.0/0 lets any SNMP manager access this community. It can be restricted to a range of IPs if desired. To configure additional communities, click Add. Figure 5 shows the specification of an SNMP V1/V2c community name of public for the read-only access of application domain status within the swlinn-poc domain.

Figure 5. SNMP Community Settings specifying and application domain

T he Trap and Notification Targets tab lets you specify the IP and port of the SNMP manager that will receive SNMP alerts and notifications. The default is UDP port 162. The community name and the SNMP version (1, 2c, or 3) must be specified. If Version 3 is used, a DataPower user name is provided in the Security Name field. This user will be configured with SNMP V3 credentials. The specific events that are alerted are configured on either the SNMP Trap Event Subscription tab or on the subscription configuration of a SNMP logging target. Events preconfigured by default on the SNMP Trap Event Subscription tab are critical device-specific events, such as memory exhaustion, or hardware issues with the power supplies, battery, or fans. To configure additional notification targets, click Add . Figure 6 shows the configuration of the recipient of SNMP alerts using SNMP Version 2c with the community name of public:

Figure 6. SNMP Trap and Notification Targets

Finally, the SNMPV3 Contexts tab gives SNMPV3 managers access to non-default application domains. To allow only SNMP polling, enabling the SNMP settings and providing a SNMPV1/V2c community is all that is required. Trap and notification targets and event subscriptions are required in sending event alerts to an SNMP manager.

As previously mentioned, some status data such as fan speeds and CPU utilization is specific to the device. Other status data such as transaction rates are segmented by application domain and are accumulated only if the statistics setting configuration is enabled, as shown in Figure 7 below. Enabling statistics has a very small impact on system utilization. Adjusting the Load Interval (the frequency of SNMP polling) will further limit this impact.

Figure 7. Statistics enabled per domain

Here is an example of a poll of an appliance metric: An SNMP manager issues a SNMP GET command for the dpStatusMemoryStatusUsage metric, which returns a scalar value of the percentage of memory being utilized. Many SNMP managers, when configured with the DataPower MIBs, provide a tree hierarchy of the status MIB from which the appropriate metric can be selected, the metric polled, and the value displayed.

Application monitoring can also be polled if the application domain is specified in the DataPower SNMP configuration. Depending on the application configuration, specific metrics can be polled to provide data on the health or throughput of the application. These application-related table entries differ from system-level metrics in that they are dynamic and are based on the key fields of these tables. For an example of a poll of an application metric, consider the dpStatusHTTPTransactions2Table table, which contains the transaction rates for all services in a domain over various time intervals. Metrics in this table are based upon the service class, such as XMLFirewallService, and the service name, such as Loopback_FW.

In addition to the event subscriptions that you can specify in the SNMP settings, you can also configure a DataPower logging target to produce SNMP logging events, which enables DataPower to send SNMP alerts for specific events of interest. Select Manage Logging Targets from the left navigation of the DataPower Web GUI from the Administration menu under the Miscellaneous heading. Click Add to create a new logging target, and specify Target Type to be SNMP. Figure 8 shows a log target with an SNMP Target Type:

Figure 8. SNMP logging target

The SNMP logging target can subscribe to and filter events just like any other DataPower logging target. The SNMP configuration’s list of trap and notification event codes specifies most critical events. An SNMP logging target in the default domain that subscribed to all events with a severity of critical or above is a similar way to produce these alerts. However, the logging target subscriptions in an application domain are more application specific. For example, you can specify logs with an MQ or SSL log category at the error or above level. You can also specify log messages generated by custom stylesheets using custom log categories. Figure 9 shows the subscription of all critical events for this SNMP type log target:

Figure 9. Logging Target Subscriptions

Now that the steps to configure and enable SNMP alerts have been described, here is a demonstration of a power supply alert. With the above configuration, the plug from one of two power supplies is pulled. Figure 10 shows log entries associated with a power supply failure:

Figure 10. System Log Entries

The SNMP configuration specified no restrictions on the SNMP Managers that could receive alerts from this appliance’s public community. Any SNMP manager listening for alerts from this appliance on Port 162 will receive a trap for the power failure event.

This section has shown you how to configure DataPower to enable monitoring of both appliance and application metrics from SNMP tools, and how to produce SNMP alerts from within an appliance. A logging target configuration was configured to produce alerts based on logging events. SNMP configuration was configured to produce alerts by subscribing to systems events (such as «Out of memory» or «Power supply has failed») as well as an application event (an SSL certificate expiration warning). Enabling statistics for application-level metrics was also shown. A poll of the memory metrics was shown to demonstrate monitoring of device metrics, and a poll of the transaction rate table was shown to demonstrate monitoring of application-specific metrics. Finally, an example of a power supply failure was used to demonstrate SNMP alerting.

What to monitor

Monitoring accomplishes multiple goals. The general health of the device and of its various physical components can be ascertained by environmental status information such as temperatures, fan speeds, and the status of batteries and power supplies. System load can be gauged by a special status value known as System Usage, in addition to more familiar measurements such as CPU, memory, and file system utilization. The amount of data being processed by the device can be determined by analyzing network interface consumption. The following section discusses several informative status values. Each section shows how to determine the data from the Web GUI, the element from the XMI response, the CLI command to execute to show the status, and the object from the SNMP Enterprise MIB that contains the value.

General device health and activity monitors

General health and activity monitors ensure that the DataPower device is operating within predefined system parameters. You can analyze system capacity via system load and CPU utilization. You can evaluate uptime to ensure that the device has not experienced an unexpected restart. Fans and temperatures are checked to avoid overheating, which can take a device out of service. The following monitors are involved in these tasks:

System usage

System Usage is a measurement of the device’s ability to accept additional work. It is a formulaic calculation based on various components of system load. System Usage is typically considered the best single indicator of overall system capacity. While it may sometimes spike to 100%, typical values are less than 75%. The secondary work list value is a calculation of queued tasks, and is of lesser interest in typical monitoring situations.

Figure 11. System Usage Status

CPU Usage statistics are provided over five time intervals. Many customers are accustomed to monitoring CPU utilization, but this metric in DataPower is not as reliable as System Usage in determining device capacity. DataPower is self-optimizing, and spikes in CPU unassociated with traffic levels may occur as the device performs background activities. CPU usage may sometimes spike all the way up to 100%, but this level is not necessarily a concern unless it is sustained over numerous consecutive polls.

Figure 12. CPU Usage Status
Memory usage

Memory Usage statistics are provided for various classifications of the appliance’s flash memory. Statistics include a percentage of total memory utilized; bytes of total, used, and free memory; and of lesser interest in typical monitoring, request, XG4, and held memory. The percentage of used memory depends on the application, the size of request and response messages, and the volume and latency of requests. Typical utilization runs less than 80%, and statistics beyond this threshold are of concern. You can use the device’s Throttle Settings to temporarily slow down request processing or to perform a warm restart, which recaptures memory in this situation.

The following system error codes are associated with these sensors and can be used to trigger alerts from the SNMP Trap Event Subscription configuration:

0x01a40001 Throttling connections due to low memory 0x01a30002 Restart due to low memory 0x01a30003 Memory usage recovered above threshold

Figure 13. Memory Usage Status
File system information

File system statistics are provided for free and total space of the encrypted, temporary, and internal file systems. Monitor all free space metrics — levels below 20% of the total space are a concern. You can use the device’s Throttle Settings to temporarily slow down request processing or to perform a warm restart, which recaptures file system space in situations of reduced free space.

The following system error codes are associated with these sensors and can be used to trigger alerts from the SNMP Trap Event Subscription configuration:

0x01a40005 Throttling connections due to low temporary file space 0x01a30006 Restart due to low temporary file space 0x01a50007 Temporary file space recovered above threshold

Figure 14. File system Usage Status
System up time

System up time indicates the elapsed time since the device was last restarted, including controlled firmware reloads as well as any unexpected device restarts. The DataPower device restarts itself automatically in conjunction with throttle configurations such as memory or file system constraints. While you can use SNMP notification for alerting, monitoring uptime via polling ensures that any notification delivery failure will not obscure these events.

Figure 15. Date and time status
Temperature sensors

Various temperature readings are available for CPUs, Memory, and System. Each has a warning and danger temperature associated with it and a status value of OK or FAIL. Monitoring the status ensures that the device is operating within the specified range. Investigate temperatures outside the ranges by checking fan speeds, airflow around device, and if necessary by contacting DataPower Support.

Figure 16. Temperature sensors status
Fan sensors

Proper functioning of the device’s fans is vital for proper operation. There are two hot swappable fan trays. If the device contains the optional hard disk drives, it will have two additional fans. Each value is associated with a minimum range and a status indicator. Monitoring the status value will ensure proper functioning of the fans. The following system error codes are associated with these sensors and can be used to trigger alerts from the SNMP Trap Event Subscription configuration:

0x02240002 Internal cooling fan has slowed 0x02220003 Internal cooling fan has stopped

Figure 17. Fan sensors status
Other sensors

There are several additional sensors grouped into the Other classification, including battery, hard disk, and power supply indicators. The intrusion detection sensor is also in this list, and it is triggered when tampering of the physical device is detected. All of these variables include a status value. Monitoring the status value will ensure proper functioning of the fans and other components.

The following system error codes are associated with these sensors and can be used to trigger alerts from the SNMP Trap Event Subscription configuration:

0x02220001 Power supply failure 0x02220004 System battery missing 0x02220005 System battery failed

Replace the battery every two years — critical level log records will begin to appear before that.

Figure 18. Other sensors status

Interface utilization statistics

Interface utilization monitors provide an analysis of the amount of data that is being received and transmitted by the DataPower device. Each device contains four gigabit interfaces. Monitoring this utilization can help you understand your transmission rates and how they change over time. Knowing that a service is increasing 10% per month can be used to anticipate additional support resources such as DataPower or backend devices.

Ethernet interfaces

Receive and transmit throughput

Figure 19. Ethernet interface status

Receive and transmit throughput information can help you understand the amount of data being processed by the device. These statistics are provided for five time values ranging from 10 seconds up to the most recent 24 hour period. This data point is an important one to capture in order to understand the network load that is being applied to the device. It includes management traffic. If you have not seperated management traffic such as Web GUI, CLI, and XMI to a separate interface, then this data will be included with any application traffic.

Each DataPower configuration (or application if you prefer) will vary significantly in terms of the processing done on individual messages. In some instances, small messages may trigger significant processing, perhaps requesting additional data from off box endpoints, performing processor intensive cryptographic operations, or in some other way generating significant system load. In another instance, large messages may be simply routed and require less processing. While there is no hard and fast rule, over time, observations of increases in data will correspond to increases in utilization of DataPower resources. Knowing this information before bottlenecks occur and alleviating it with additional DataPower devices can help you avoid system interruptions.

Figure 20. Rx throughput status
HTTP Connections

HTTP connections are produced at the domain level. Statistics must be enabled for each domain that is to produce HTTP connection data. One peculiarity is that HTTP connection data is not accumulated for services in loopback mode. The status data is segmented by XML-Manager and contains information about HTTP connections, such as request and reuse. This data can help you understand the level of connections and can be used to judge utilization growth over time.

Figure 21. HTTP connections status

Transaction rates and elapsed times for individual services are accumulated at the domain and within domain service level. Transaction rate and time are not provided unless statistics are enabled for each domain. This data can help you understand the number of transactions processed and the average response time of those transactions for a particular service over a number of time intervals.

Transaction rate and time

Figure 22. Transaction rate status

Other network status providers

DataPower supports many protocols beyond the HTTP examples discussed so far, including support for FTP, IMS, MQ, NFS, NTP, SQL, Tibco, and WebSphere JMS. Each of these protocols is represented by status providers, and as in the case of the previous examples, each is supported by the Web GUI, CLI, XMI, and SNMP. Individual configurations may not use any of these additional protocols, and few will use all of them. However, in a configuration that is using one or more of these protocols, monitoring the related status provider is prudent.

Best practices

Successful monitoring of the DataPower appliance will utilize active and proactive inquiry of status information. Configuration of SNMP tools will require listening for traps sent by the device and periodic polling of the device for MIB status data. These actions require a combination of DataPower SNMP Trap Event Subscription configuration and configuration of the SNMP monitoring agent in polling and potentially based on returned status values.

In addition to device monitoring, application monitoring is also a useful practice. In this instance sample messages may be sent from robotic clients through the DataPower service to ensure that all network links (including load balancers) are operational. In some instances, this effort is extended to include sending messages through to backend service provider applications to ensure that both frontside and backside links are in service. Both DataPower and backside resources must be configured to respond appropriately to these test messages.

The DataPower SMMP trap subscription capability is a useful method of leveraging SNMP notification of events within DataPower. Here is a suggested list of error codes to subscribe to. In the event that the error is produced, the SNMP agent on DataPower will send an Alert/Trap to the SNMP monitor.

Suggested error code subscription

MIB status values to monitor

It is recommended that SNMP monitors be configured to fetch and report on the following conditions:

MIB status values to monitor for interface utilization

In addition to polling and inquiring of data, it is important to ascertain the normal traffic patterns of applications over time. The best way to do this is to capture and monitor the amount of network traffic that the device is processing. The transmit and receive values below will help you predict when devices will become saturated with traffic. Knowing this ahead of time can help you avoid service disruptions.

Conclusion

Best practice monitoring of DataPower is a three-pronged activity:

• Continuously verify the status of the DataPower environment through polling status data and subscribing to SNMP traps.
• Monitor device utilization and capacity through analysis of system usage data and interpretation of Ethernet activity.
• Perform complete application path verification by sending test message through the DataPower service configuration and perhaps on through to backend resources.

Performing these three actions will ensure that services are available and the DataPower appliance is performing within standard ranges of operation.

Acknowledgements

The authors wish to thank all those who participated in the development of this developerWorks article. Of special note are the contributions of Shiu-Fun Poon, Matthias Seibler, and Gaurang Shah of WebSphere DataPower Engineering, and Bill Hines of WebSphere DataPower Technical Sales.

• WebSphere DataPower SOA Appliances developer resources page
  Technical resources to help you use WebSphere DataPower SOA Appliances to simplify, secure, and accelerate XML and Web services deployments within an SOA.
• WebSphere DataPower SOA Appliances product page
  Product descriptions, product news, training information, support information, and more.
• WebSphere DataPower SOA Appliances product library
  Product announcements, case studies, white papers, and more.
• WebSphere DataPower SOA Appliances documentation
  Complete documentation for the DataPower XA35, XS40, XI50, XB60, and XM70 Appliances.
• WebSphere DataPower SOA Appliances support
  A searchable database of support problems and their solutions, plus downloads, fixes, problem tracking, and more.
• WebSphere DataPower SOA Appliance Handbook
  This retail book shows you how to use DataPower Appliances from the network, security, and ESB perspectives. The book describes installation, configuration, management, monitoring, configuration, build, deployment, DataPower as a network device, and DataPower services, especially the «big three» of XML Firewall, Web Service Proxy, and Multi-Protocol Gateway.
• IBM Redbook: IBM WebSphere DataPower SOA Appliances, Part I: Overview and getting started
  DataPower SOA appliances are purpose-built, easy-to-deploy network devices that simplify, secure, and accelerate your XML and Web services deployments while extending your SOA infrastructure. This IBM Redbook describes DataPower architecture, use cases, deployment scenarios, and implementation details, as well as best practices for SOA message-oriented architecture in a production ESB legacy environment.
• IBM Redbook: IBM WebSphere DataPower SOA Appliances Part II: Authentication and Authorization
  This IBM Redbook includes the following DataPower authentication and authorization topics: basic concepts, creating policies, using Tivoli Access Manager, and using LDAP directories.
• IBM Redbook: IBM WebSphere DataPower SOA Appliances Part III: XML Security Guide
  This IBM Redbook describes how to use a DataPower appliance to secure incoming Web Services within an SOA environment, how to integrate your DataPower appliance with WebSphere Message Broker, and how to protect against security attacks by implementing the XML Denial of Service (XDoS) provided by DataPower appliances.
• IBM Redbook: IBM WebSphere DataPower SOA Appliances Part IV: Management and Governance
  This IBM Redbook describes how to integrate a DataPower appliance with other products such as WebSphere Registry and Repository, IBM Tivoli Composite Application Manager for SOA, and Tivoli Composite Application Manager System Edition.
• IBM Redpaper: WebSphere DataPower SOA Appliances: The XML Management Interface
  This IBM Redpaper describes the XML Management Interface, which is the third way to configure and administer the WebSphere DataPower SOA Appliance, in addition to the WebGUI and the CLI.
• developerWorks WebSphere application connectivity developer resources
  How-to articles, downloads, tutorials, education, product info, and other resources to help you build WebSphere application connectivity and business integration solutions.
• developerWorks WebSphere business process management developer resources
  WebSphere BPM how-to articles, downloads, tutorials, education, product info, and other resources to help you model, assemble, deploy, and manage business processes.
• developerWorks WebSphere SOA and Web services developer resources
  How-to articles, downloads, tutorials, education, product info, and other resources to help you design and build WebSphere SOA and Web services solutions.
• Most popular WebSphere trial downloads
  No-charge trial downloads for key WebSphere products.
• WebSphere on-demand demos
  Download, watch, and learn what WebSphere products and WebSphere-related technologies can do for your company.
• developerWorks WebSphere weekly newsletter
  The developerWorks newsletter gives you the latest articles and information only on those topics that interest you. In addition to WebSphere, you can select from Java, Linux, Open source, Rational, SOA, Web services, and other topics. Subscribe now and design your custom mailing.
• WebSphere-related books from IBM Press
  Convenient online ordering through Barnes & Noble. DB2 , Lotus , Rational , Tivoli , and WebSphere products. —>
• developerWorks blogs
  Join a conversation with developerWorks users and authors, and IBM editors and developers.
• developerWorks on Twitter
  Check out recent Twitter messages and URLs.