Error on amqp connection что это за ошибка
Обработка ошибок с помощью Spring AMQP
Узнайте о различных способах обработки ошибок с помощью Spring AMQP с помощью RabbitMQ.
1. введение
2. Настройка среды
Давайте запустим RabbitMQ как автономный сервер. Мы запустим его в контейнере Docker, выполнив следующую команду:
3. Сценарий Отказа
Обычно существует больше типов ошибок, которые могут возникнуть в системах обмена сообщениями по сравнению с монолитными или однопакетными приложениями из-за их распределенной природы.
Мы можем указать на некоторые типы исключений:
Конечно, этот список отказов не является исчерпывающим, но содержит наиболее распространенные типы ошибок.
В следующих разделах мы в основном сосредоточимся на специфичных для приложений и высокоуровневых ошибках, а затем рассмотрим глобальные стратегии обработки ошибок.
4. Настройка проекта
Теперь давайте определим простую конфигурацию очереди и обмена для начала:
Далее, давайте создадим простого производителя:
И, наконец, потребитель, который бросает исключение:
По умолчанию все неудачные сообщения будут немедленно запрашиваться в начале целевой очереди снова и снова.
Давайте запустим наш пример приложения, выполнив следующую команду Maven:
Теперь мы должны увидеть аналогичный результирующий результат:
Следовательно, по умолчанию мы будем видеть бесконечное количество таких сообщений в выходных данных.
Чтобы изменить это поведение у нас есть два варианта:
Теперь давайте узнаем, как обрабатывать неудачные сообщения более разумным способом.
5. Очередь Мертвых писем
Что еще более важно, это помогает предотвратить бесконечные циклы в очередях, которые постоянно обрабатывают плохие сообщения и снижают производительность системы.
Теперь давайте посмотрим, как обрабатывать исключения, применяя подход очереди мертвых писем.
5.1. Базовая конфигурация
Чтобы настроить DLQ, нам нужно указать дополнительные аргументы при определении нашей очереди:
Второй аргумент столь же важен, как и установка ключей маршрутизации для простых сообщений. Эта опция изменяет начальный ключ маршрутизации сообщения для дальнейшей маршрутизации по DLX.
5.2. Неудачная Маршрутизация Сообщений
Поэтому, когда сообщение не доставляется, оно направляется на обмен мертвыми письмами. Но, как мы уже отмечали, DLX-это нормальный обмен. Поэтому, если ключ маршрутизации неудачного сообщения не совпадает с обменом, он не будет доставлен в DLQ.
Таким образом, если мы опустим аргумент x-dead-letter-routing-key в нашем примере, неудачное сообщение застрянет в бесконечном цикле повторных попыток.
Кроме того, исходная метаинформация сообщения доступна в x-death heaven:
То приведенная выше информация доступна в консоли управления RabbitMQ обычно работает локально на порту 15672.
5.3. Обмен Мертвыми письмами
В предыдущем разделе мы видели, что ключ маршрутизации изменяется, когда сообщение направляется на обмен мертвыми буквами. Но такое поведение не всегда желательно. Мы можем изменить его, настроив DLX самостоятельно и определив его с помощью типа fanout :
Теперь, если мы запустим наш пример, неудачное сообщение должно быть доставлено в DLQ, но без изменения начального ключа маршрутизации:
5.4. Обработка Сообщений Очереди Мертвых Писем
Конечно, причина, по которой мы переместили их в очередь мертвых писем, заключается в том, что они могут быть переработаны в другое время.
Давайте определим слушателя для очереди мертвых писем:
Если мы запустим наш пример кода сейчас, то увидим вывод журнала:
Мы получили неудачное сообщение, но что нам делать дальше? Ответ зависит от конкретных системных требований, вида исключения или типа сообщения.
Например, мы можем просто запросить сообщение в исходное место назначения:
Но такая логика исключений не отличается от политики повторных попыток по умолчанию:
Общая стратегия может потребовать повторной обработки сообщения в течение n раз, а затем отклонить его. Давайте реализуем эту стратегию, используя заголовки сообщений:
5.5. Очередь на Парковку
С другой стороны, рассмотрим ситуацию, когда мы не можем просто отбросить сообщение, например, это может быть транзакция в банковской сфере. Кроме того, иногда сообщение может потребовать ручной обработки или нам просто нужно записать сообщения, которые потерпели неудачу более n раз.
Давайте теперь воплотим эту идею:
Во – вторых, давайте реорганизуем логику прослушивателя, чтобы отправить сообщение в очередь парковки:
В конце концов, нам также нужно обрабатывать сообщения, поступающие в очередь парковки:
Теперь мы можем сохранить неудачное сообщение в базе данных или, возможно, отправить уведомление по электронной почте.
Давайте проверим эту логику, запустив наше приложение:
Как видно из выходных данных, после нескольких неудачных попыток сообщение было отправлено в очередь парковки.
6. Пользовательская Обработка Ошибок
В предыдущем разделе мы видели, как обрабатывать сбои с помощью выделенных очередей и обменов. Однако иногда нам может потребоваться отловить все ошибки, например, для регистрации или сохранения их в базе данных.
6.1. Глобальный обработчик ошибок
Проще говоря, Условное отклонение ErrorHandler решает, отклонять ли конкретное сообщение или нет. Когда сообщение, вызвавшее исключение, отклоняется, оно не запрашивается.
6.2. Стратегия Фатальных Исключений
По умолчанию эти исключения фатальны:
Вместо реализации интерфейса Ierrorhandler мы можем просто предоставить нашу стратегию Фатальных исключений :
Наконец, нам нужно передать нашу пользовательскую стратегию конструктору Conditional Rejecting ErrorHandler :
7. Заключение
В этом уроке мы обсудили различные способы обработки ошибок при использовании Spring AMQP и, в частности, RabbitMQ.
Каждая система нуждается в определенной стратегии обработки ошибок. Мы рассмотрели наиболее распространенные способы обработки ошибок в архитектуре, управляемой событиями. Кроме того, мы убедились, что можем объединить несколько стратегий для создания более комплексного и надежного решения.
Error on amqp connection что это за ошибка
Development of a robust application, be it message publisher or message consumer, involves dealing with multiple kinds of failures: protocol exceptions, network failures, broker failures and so on. Correct error handling and recovery is not easy. This guide explains how the amqp gem helps you in dealing with issues like
This work is licensed under a Creative Commons Attribution 3.0 Unported License (including images and stylesheets). The source is available on Github.
Covered versions
This guide covers Ruby amqp gem 1.5.x.
Code examples
There are several examples in the git repository dedicated to the topic of error handling and recovery. Feel free to contribute new examples.
Initial broker connection failures
When applications connect to the broker, they need to handle connection failures. Networks are not 100% reliable, even with modern system configuration tools like Chef or Puppet misconfigurations happen and the broker might also be down. Error detection should happen as early as possible. There are two ways of detecting TCP connection failure, the first one is to catch an exception:
AMQP.connect (and AMQP.start ) will raise AMQP::TCPConnectionFailed if a connection fails. Code that catches it can write to a log about the issue or use retry to execute the begin block one more time. Because initial connection failures are due to misconfiguration or network outage, reconnection to the same endpoint (hostname, port, vhost combination) will result in the same issue over and over. TBD: failover, connection to the cluster.
An alternative way of handling connection failure is with an errback (a callback for specific kind of error):
If you connect to the broker from code in a class (as opposed to top-level scope in a script), Object#method can be used to pass object method as a handler instead of a Proc.
Authentication failures
Another reason why a connection may fail is authentication failure. Handling authentication failure is very similar to handling initial TCP connection failure:
Default handler
default handler raises AMQP::PossibleAuthenticationFailureError :
In case you are wondering why callback name has «possible” in it: AMQP 0.9.1 spec requires broker implementations to simply close TCP connection without sending any more data when an exception (such as authentication failure) occurs before AMQP connection is open. In practice, however, when broker closes TCP connection between successful TCP connection and before AMQP connection is open, it means that authentication has failed.
Handling network connection interruptions
Sometimes it is necessary for other entities in an application to react to network failures. amqp gem 0.8.0 and later provides a number of event handlers to make this task easier for developers. This set of features is known as the «shutdown protocol” (the word «protocol” here means «API interface” or «behavior”, not network protocol).
The Shutdown protocol revolves around two events:
In this section, we will concentrate on the former. When a network connection fails, the underlying networking library detects it and runs a piece of code on AMQP::Session to handle it. That, in turn, propagates this event to channels, channels propagate it to exchanges and queues, queues propagate it to their consumers (if any). Each of these entities in the object graph can react to network interruption by executing application-defined callbacks.
Shutdown Protocol methods on AMQP::Session
The difference between these methods is that AMQP::Session#on_tcp_connection_loss is used to define a callback that will be executed once when TCP connection fails. It is possible that reconnection attempts will not succeed immediately, so there will be subsequent failures. To react to those, AMQP::Session#on_connection_interruption method is used.
The first argument that both of these methods yield to the handler that your application defines is the connection itself. This is done to make sure that you can register Ruby objects as handlers, and they do not have to keep any state around (for example, connection instances):
Note that AMQP::Session#on_connection_interruption callback is called before this event is propagated to channels, queues and so on.
Different applications handle connection failures differently. It is very common to use AMQP::Session#reconnect method to schedule a reconnection to the same host, or use AMQP::Session#reconnect_to to connect to a different one. For some applications it is OK to simply exit and wait to be restarted at a later point in time, for example, by a process monitoring system like Nagios or Monit.
Shutdown Protocol methods on AMQP::Channel
Note that AMQP::Channel#on_connection_interruption callback is called after this event is propagated to exchanges, queues and so on. Right after that channel state is reset, except for error handling/recovery-related callbacks.
Many applications do not need per-channel network failure handling.
Shutdown Protocol methods on AMQP::Exchange
Many applications do not need per-exchange network failure handling.
Shutdown Protocol methods on AMQP::Queue
Note that AMQP::Queue#on_connection_interruption callback is called after this event is propagated to consumers.
Many applications do not need per-queue network failure handling.`
Shutdown Protocol methods on AMQP::Consumer
Many applications do not need per-consumer network failure handling.`
Recovering from network connection failures
Detecting network connections is nearly useless if an AMQP-based application cannot recover from them. Recovery is the hard part in «error handling and recovery”. Fortunately, the recovery process for many applications follows one simple scheme that the amqp gem can perform automatically for you.
The recovery process, both manual and automatic, always begins with re-opening an AMQP connection and then all the channels on that connection.`
Manual recovery
Similarly to the Shutdown Protocol, the amqp gem entities implement a Recovery Protocol. The Recovery Protocol consists of three methods that connections, channels, queues, consumers and exchanges all implement:
AMQP::Session#before_recovery lets application developers register a callback that will be executed after TCP connection is re-established but before AMQP connection is reopened. AMQP::Session#after_recovery is similar except that the callback is run after AMQP connection is reopened.
Recovery process for AMQP applications usually involves the following steps:
Automatic recovery
Many applications use the same recovery strategy that consists of the following steps:
The amqp gem provides a feature known as «automatic recovery” that is opt-in (not opt-out, not used by default) and lets application developers get the aforementioned recovery strategy by setting one additional attribute on AMQP::Channel instance:
A more verbose way to do the same thing:
Auto recovery mode can be turned on and off any number of times during channel life cycle, although a very small percentage of applications actually do this. Typically you decide what channels should be using automatic recovery during the application design stage.
Full example (run it, then shut down AMQP broker running on localhost, then bring it back up and use management tools such as rabbitmqctl to see that queues, bindings and consumers have all recovered):
Server-named queues, when recovered automatically, will get new server-generated names to guarantee there are no name collisions.
When in doubt, try using automatic recovery first. If it is not sufficient for your application, switch to manual recovery using events and callbacks introduced in the «Manual recovery” section.
Detecting broker failures
AMQP applications see broker failure as TCP connection loss. There is no reliable way to know whether there is a network problem or a network peer is down.
AMQP connection-level exceptions
Handling connection-level exceptions
Connection-level exceptions are rare and may indicate a serious issue with a client library or in-flight data corruption. The AMQP 0.9.1 specification mandates that a connection that has errored cannot be used any more and must be closed. In any case, your application should be prepared to handle this kind of error. To define a handler, use AMQP::Session#on_error method that takes a callback and yields two arguments to it when a connection-level exception happens:
Status codes are similar to those of HTTP. For the full list of error codes and their meaning, consult AMQP 0.9.1 constants reference.
Only one connection-level exception handler can be defined per connection instance (the one added last replaces previously added ones).
Handling graceful broker shutdown
When an AMQP broker is shut down, it properly closes connections first. To do so, it uses connection.close AMQP method. AMQP clients then need to check if the reply code is equal to 320 (CONNECTION_FORCED) to distinguish graceful shutdown. With RabbitMQ, when broker is stopped using
reply_text will be set to
Each application chooses how to handle graceful broker shutdowns individually, so amqp gem’s automatic reconnection does not cover graceful broker shutdowns. Applications that want to reconnect when broker is stopped can use AMQP::Session#periodically_reconnect like so:
Once AMQP connection is re-opened, channels in automatic recovery mode will recover just like they do after network outages.
Integrating channel-level exceptions handling with object-oriented Ruby code
Error handling can be easily integrated into object-oriented Ruby code (in fact, this is highly encouraged). A common technique is to combine Object#method and Method#to_proc and use object methods as error handlers:
Full example that uses objects:
AMQP channel-level exceptions
Handling channel-level exceptions
Channel-level exceptions are more common than connection-level ones. They are handled in a similar manner, by defining a callback with AMQP::Channel#on_error method that takes a callback and yields two arguments to it when a channel-level exception happens:
Status codes are similar to those of HTTP. For the full list of error codes and their meaning, consult AMQP 0.9.1 constants reference.
Only one channel-level exception handler can be defined per channel instance (the one added last replaces previously added ones).`
Integrating channel-level exceptions handling with object-oriented Ruby code
Error handling can be easily integrated into object-oriented Ruby code (in fact, this is highly encouraged).A common technique is to combine Object#method and Method#to_proc and use object methods as error handlers. For example of this, see section on connection-level exceptions above.
Because channel-level exceptions may be raised because of multiple unrelated reasons and often indicate misconfigurations, how they are handled isvery specific to particular applications. A common strategy is to log an error and then open and use another channel.
Common channel-level exceptions and what they mean
A few channel-level exceptions are common and deserve more attention.
406 Precondition Failed
405 Resource Locked
403 Access Refused
Conclusion
Distributed applications introduce a whole new class of failures developers need to be aware of. Many of them stem from unreliable networks. The famous Fallacies of Distributed Computing list common assumptions software engineers must not make:
Unfortunately, applications that use Ruby and AMQP are not immune to these problems and developers need to always keep that in mind. This list is just as relevant today as it was in the 90s.
Ruby amqp gem 0.8.x and later lets applications define handlers that handle connection-level exceptions, channel-level exceptions and TCP connection failures. Handling AMQP exceptions and network connection failures is relatively easy. Re-declaring AMQP instances that the application works with is where most of the complexity comes from. By using Ruby objects as error handlers, the declaration of AMQP entities can be done in one place, making code much easier to understand and maintain.
amqp gem error and interruption handling is not a copy of RabbitMQ Java client’s Shutdown Protocol, but they turn out to be similar with respect to network failures and connection-level exceptions.
Tell Us What You Think!
Please take a moment to tell us what you think about this guide on Twitter or the RabbitMQ mailing list
Let us know what was unclear or what has not been covered. Maybe you do not like the guide style or grammar or discover spelling mistakes. Reader feedback is key to making the documentation better.
This website was developed by the Ruby AMQP Team.
RabbitMQ: рукопожатие прекращено сервером (ОТКАЗ ОТ ДОСТУПА)
Я пытаюсь отправить сообщения RabbitMQ с моего хост-компьютера на экземпляр Minikube с развернутым кластером RabbitMQ.
При запуске моего сценария отправки я получаю эту ошибку:
В лог-файлах брокера я вижу такую строку:
Я уверен, что у меня есть правильные учетные данные, поскольку я получил их непосредственно из модуля RabbitMQ, следуя официальной документации (ссылка).
Мой сценарий отправки приведен ниже:
Я знаю, что код работает, поскольку я запустил тот же сценарий для моей настройки localhost, и он сработал. Единственное, что я изменил, это URL-адрес (для сервиса Minikube RabbitMQ).
Я видел несколько других сообщений, содержащих аналогичную проблему, но большинство решений связано с включением правильных учетных данных в URI, что я и сделал.
1 ответ
Вы можете использовать переадресацию порта для службы rabbitMQ на локальный компьютер и использовать вход в пользовательский интерфейс и проверять пароль с помощью пользовательского интерфейса, предоставленного самим RabbitMQ.
Должно быть достаточно
Для оформления вы можете проверить, можете ли вы войти в систему из самого контейнера. В случае сбоя входа в систему из контейнера вы также можете проверить файл yaml или схему управления, которую вы используете, на предмет методов входа и учетных данных. Простой вход может быть отключен.
Другая ситуация может быть с раздачей. При развертывании RabbitMQ я стараюсь использовать графики битнами. Я могу их предложить.
Если все это не помогает, вы можете использовать другой способ. Вы можете попробовать создать другого пользователя с правами администратора для подключения к RabbitMQ, а затем продолжать его использовать.
Для получения дополнительной информации вы можете опубликовать журналы контейнеров / подов, чтобы мы их увидели. Добрый день.
Troubleshooting Network Connectivity
Overview
This guide accompanies the one on networking and focuses on troubleshooting of network connections.
For connections that use TLS there is an additional guide on troubleshooting TLS.
Troubleshooting Methodology
Troubleshooting of network connectivity issues is a broad topic. There are entire books written about it. This guide explains a methodology and widely available networking tools that help narrow most common issues down efficiently.
Networking protocols are layered. So are problems with them. An effective troubleshooting strategy typically uses the process of elimination to pinpoint the issue (or multiple issues), starting at higher levels. Specifically for messaging technologies, the following steps are often effective and sufficient:
These steps, when performed in sequence, usually help identify the root cause of the vast majority of networking issues. Troubleshooting tools and techniques for levels lower than the Internet (networking) layer are outside of the scope of this guide.
Certain problems only happen in environments with a high degree of connection churn. Client connections can be inspected using the management UI. It is also possible to inspect all TCP connections of a node and their state. That information collected over time, combined with server logs, will help detect connection churn, file descriptor exhaustion and related issues.
Verify Client Configuration
All developers and operators have been there: typos, outdated values, issues in provisioning tools, mixed up public and private key paths, and so on. Step one is to double check application and client library configuration.
Verify Server Configuration
Verifying server configuration helps prove that RabbitMQ is running with the expected set of settings related to networking. It also verifies that the node is actually running. Here are the recommended steps:
Note that in older RabbitMQ versions, the status and environment commands were only available as part of rabbitmqctl: rabbitmqctl status and so on. In modern versions either tool can be used to run those commands but rabbitmq-diagnostics is what most documentation guides will typically recommend.
The listeners section will look something like this:
In the above example, there are 6 TCP listeners on the node:
In second example, there are 4 TCP listeners on the node:
All listeners are bound to all available interfaces.
Inspecting TCP listeners used by a node helps spot non-standard port configuration, protocol plugins (e.g. MQTT) that are supposed to be configured but aren’t, cases when the node is limited to only a few network interfaces, and so on. If a port is not on the listener list it means the node cannot accept any connections on it.
Inspect Server Logs
RabbitMQ nodes will log key client connection lifecycle events. A TCP connection must be successfully established and at least 1 byte of data must be sent by the peer for a connection to be considered (and logged as) accepted.
From this point, connection handshake and negotiation proceeds as defined by the specification of the messaging protocol used, e.g. AMQP 0-9-1, AMQP 1.0 or MQTT.
If no events are logged, this means that either there were no successful inbound TCP connections or they sent no data.
Hostname Resolution
It is very common for applications to use hostnames or URIs with hostnames when connecting to RabbitMQ. dig and nslookup are commonly used tools for troubleshooting hostnames resolution.
Port Access
Besides hostname resolution and IP routing issues, TCP port inaccessibility for outside connections is a common reason for failing client connections. telnet is a commonly used, very minimalistic tool for testing TCP connections to a particular hostname and port.
Since 12345 is not a correct AMQP 0-9-1 or AMQP 1.0 protocol header, so the server closes TCP connection:
After telnet connection succeeds, use Control + ] and then Control + D to quit it.
Failed or timing out telnet connections strongly suggest there’s a proxy, load balancer or firewall that blocks incoming connections on the target port. It could also be due to RabbitMQ process not running on the target node or uses a non-standard port. Those scenarios should be eliminated at the step that double checks server listener configuration.
There’s a great number of firewall, proxy and load balancer tools and products. iptables is a commonly used firewall on Linux and other UNIX-like systems. There is no shortage of iptables tutorials on the Web.
Open ports, TCP and UDP connections of a node can be inspected using netstat, ss, lsof.
The following example uses lsof to display OS processes that listen on port 5672 and use IPv4:
Similarly, for programs that use IPv6:
If the above commands produce no output then no local OS processes listen on the given port.
The following example uses ss to display listening TCP sockets that use IPv4 and their OS processes:
Similarly, for TCP sockets that use IPv6:
For the list of ports used by RabbitMQ and its various plugins, see above. Generally all ports used for external connections must be allowed by the firewalls and proxies.
rabbitmq-diagnostics listeners and rabbitmqctl status can be used to list enabled listeners and their ports on a RabbitMQ node.
IP Routing
Messaging protocols supported by RabbitMQ use TCP and require IP routing between clients and RabbitMQ hosts to be functional. There are several tools and techniques that can be used to verify IP routing between two hosts. traceroute and ping are two common options available for many operating systems. Most routing table inspection tools are OS-specific.
Note that both traceroute and ping use ICMP while RabbitMQ client libraries and inter-node connections use TCP. Therefore a successful ping run alone does not guarantee successful client connectivity.
Both traceroute and ping have Web-based and GUI tools built on top.
Capturing Traffic
All network activity can be inspected, filtered and analyzed using a traffic capture.
tcpdump and its GUI sibling Wireshark are the industry standards for capturing traffic, filtering and analysis. Both support all protocols supported by RabbitMQ. See the Using Wireshark with RabbitMQ guide for an overview.
TLS Connections
For connections that use TLS there is a separate guide on troubleshooting TLS.
When adopting TLS it is important to make sure that clients use correct port to connect (see the list of ports above) and that they are instructed to use TLS (perform TLS upgrade). A client that is not configured to use TLS will successfully connect to a TLS-enabled server port but its connection will then time out since it never performs the TLS upgrade that the server expects.
A TLS-enabled client connecting to a non-TLS enabled port will successfully connect and try to perform a TLS upgrade which the server does not expect, this triggering a protocol parser exception. Such exceptions will be logged by the server.
Inspecting Connections
Open ports, TCP and UDP connections of a node can be inspected using netstat, ss, lsof.
The following example uses netstat to list all TCP connection sockets regardless of their state and interface. IP addresses will be displayed as numbers instead of being resolved to domain names. Program names will be printed next to numeric port values (as opposed to protocol names).
Both inbound (client, peer nodes, CLI tools) and outgoing (peer nodes, Federation links and Shovels) connections can be inspected this way.
Combining connection information from management UI or CLI tools with those of netstat or ss can help troubleshoot misbehaving applications, application instances and client libraries.
Most relevant connection metrics can be collected, aggregated and monitored using Prometheus and Grafana.
Detecting High Connection Churn
High connection churn (lots of connections opened and closed after a brief period of time) can lead to resource exhaustion. It is therefore important to be able to identify such scenarios. netstat and ss are most popular options for inspecting TCP connections. A lot of connections in the TIME_WAIT state is a likely symptom of high connection churn. Lots of connections in states other than ESTABLISHED also might be a symptom worth investigating.
Evidence of short lived connections can be found in RabbitMQ log files. E.g. here’s an example of such connection that lasted only a few milliseconds:
Getting Help and Providing Feedback
If you have questions about the contents of this guide or any other topic related to RabbitMQ, don’t hesitate to ask them on the RabbitMQ mailing list.
Help Us Improve the Docs <3
If you’d like to contribute an improvement to the site, its source is available on GitHub. Simply fork the repository and submit a pull request. Thank you!