Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Investigate 'Operation not permitted' messages #1

Open
ralphlange opened this issue Aug 9, 2018 · 5 comments
Open

Investigate 'Operation not permitted' messages #1

ralphlange opened this issue Aug 9, 2018 · 5 comments

Comments

@ralphlange
Copy link

Are you sure these messages are related to the iptables trick?

They are pointing to some configuration issue, but the iptables rules are related to incoming UDP messages, while these errors are caused by trouble with outgoing UDP messages.
@jeonghanlee
Copy link
Owner

I am not sure these messages are related to the iptables trick :)

  • IOC 1, IOC2 and IOC 3 are running on a Linux host
  • Three camonitor commands (camonitor pv1 pv2 pv3) are running on three difference clients in the different Linux machines

That happens when I close one camonitor command in a client and re-run it immediately (less than 1 sec). Then, few (not sure) IOCs return these messages. I will test few possible scenario later, and let you know.

@jeonghanlee
Copy link
Owner

jeonghanlee commented Aug 10, 2018 •
edited

@ralphlange

It only happens when I try to get PVs with one command, for example,

caget pv1 pv2 pv3

or

camonitor pv1 pv2 pv3

If I call them individually as follow:

caget pv1
caget pv2
caget pv3

in infinite while loop, I cannot see the message CAS: UDP send to 10.4.8.11:35593 failed - Operation not permitted

tcpdump results

  • case 1 (caget pv1, caget pv2, caget pv3)
17:57:56.029422 IP 10.4.8.11.37818 > 10.0.6.60.5064: UDP, length 64
17:57:56.031685 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [S], seq 2684663385, win 29200, options [mss 1460,sackOK,TS val 26131167 ecr 0,nop,wscale 7], length 0
17:57:56.031764 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [S.], seq 4077782817, ack 2684663386, win 28960, options [mss 1460,sackOK,TS val 318196956 ecr 26131167,nop,wscale 7], length 0
17:57:56.032534 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [.], ack 1, win 229, options [nop,nop,TS val 26131167 ecr 318196956], length 0
17:57:56.032799 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [P.], seq 1:121, ack 1, win 229, options [nop,nop,TS val 26131167 ecr 318196956], length 120
17:57:56.032845 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [.], ack 121, win 227, options [nop,nop,TS val 318196956 ecr 26131167], length 0
17:57:56.032919 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [P.], seq 1:49, ack 121, win 227, options [nop,nop,TS val 318196956 ecr 26131167], length 48
17:57:56.034058 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [.], ack 49, win 229, options [nop,nop,TS val 26131167 ecr 318196956], length 0
17:57:56.034119 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [P.], seq 121:137, ack 49, win 229, options [nop,nop,TS val 26131168 ecr 318196956], length 16
17:57:56.034245 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [P.], seq 49:89, ack 137, win 227, options [nop,nop,TS val 318196956 ecr 26131168], length 40
17:57:56.036141 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [P.], seq 137:153, ack 89, win 229, options [nop,nop,TS val 26131168 ecr 318196956], length 16
17:57:56.036170 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [F.], seq 153, ack 89, win 229, options [nop,nop,TS val 26131168 ecr 318196956], length 0
17:57:56.036244 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [P.], seq 89:105, ack 154, win 227, options [nop,nop,TS val 318196957 ecr 26131168], length 16
17:57:56.036431 IP 10.0.6.60.5064 > 10.4.8.11.47288: Flags [F.], seq 105, ack 154, win 227, options [nop,nop,TS val 318196957 ecr 26131168], length 0
17:57:56.037114 IP 10.4.8.11.47288 > 10.0.6.60.5064: Flags [.], ack 106, win 229, options [nop,nop,TS val 26131168 ecr 318196957], length 0
17:57:56.068479 IP 10.4.8.11.54430 > 10.0.6.60.5064: UDP, length 64
17:57:56.109014 IP 10.4.8.11.45956 > 10.0.6.60.5064: UDP, length 64
  • case 2 (caget pv1 pv2 pv3)
17:52:48.297983 IP 10.4.8.11.35593 > 10.0.6.60.5064: UDP, length 160
17:52:48.329445 IP 10.4.8.11.35593 > 10.0.6.60.5064: UDP, length 64
17:52:48.331081 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [S], seq 535013640, win 29200, options [mss 1460,sackOK,TS val 26054242 ecr 0,nop,wscale 7], length 0
17:52:48.331160 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [S.], seq 1709813222, ack 535013641, win 28960, options [mss 1460,sackOK,TS val 318120031 ecr 26054242,nop,wscale 7], length 0
17:52:48.332023 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [.], ack 1, win 229, options [nop,nop,TS val 26054242 ecr 318120031], length 0
17:52:48.332230 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [P.], seq 1:121, ack 1, win 229, options [nop,nop,TS val 26054242 ecr 318120031], length 120
17:52:48.332270 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [.], ack 121, win 227, options [nop,nop,TS val 318120031 ecr 26054242], length 0
17:52:48.332438 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [P.], seq 1:49, ack 121, win 227, options [nop,nop,TS val 318120031 ecr 26054242], length 48
17:52:48.333255 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [.], ack 49, win 229, options [nop,nop,TS val 26054242 ecr 318120031], length 0
17:52:48.334186 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [P.], seq 121:137, ack 49, win 229, options [nop,nop,TS val 26054242 ecr 318120031], length 16
17:52:48.334314 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [P.], seq 49:89, ack 137, win 227, options [nop,nop,TS val 318120031 ecr 26054242], length 40
17:52:48.336255 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [P.], seq 137:153, ack 89, win 229, options [nop,nop,TS val 26054243 ecr 318120031], length 16
17:52:48.336311 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [F.], seq 153, ack 89, win 229, options [nop,nop,TS val 26054243 ecr 318120031], length 0
17:52:48.336423 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [P.], seq 89:105, ack 154, win 227, options [nop,nop,TS val 318120032 ecr 26054243], length 16
17:52:48.336608 IP 10.0.6.60.5064 > 10.4.8.11.47258: Flags [F.], seq 105, ack 154, win 227, options [nop,nop,TS val 318120032 ecr 26054243], length 0
17:52:48.337295 IP 10.4.8.11.47258 > 10.0.6.60.5064: Flags [.], ack 106, win 229, options [nop,nop,TS val 26054243 ecr 318120032], length 0

In this case, I see the following message in IOC 1
CAS: UDP send to 10.4.8.11:35593 failed - Operation not permitted

What do you think? Can we improve the iptable trick?

@jeonghanlee
Copy link
Owner

Since last weekend with case1

IOC1:Test:IocStats:UPTIME      2 days, 16:19:58

with the infinite caget calls as

caget pv1
caget pv2
caget pv3

there is no message in three IOCs. It looks like the whole condition of the iptables trick doesn't cover the following case:

caget pv1 pv2  pv3

Then, I guess, it may be related with how caget with n pv variables connect to an IOC.

@ralphlange
Copy link
Author

The iptables rule changes the destination address of incoming UDP packets (name resolution requests) from the unicast address of the local host to the broadcast address. It does not look at the content of packets.
This trick allows the client to reach all IOCs on the host. It does work in your case, doesn't it?

The IOC error message is printed when the IOC wants to send a UDP packet (name resolution answer).

In your tcp dumps:
In the first case you see three UDP messages of length 64, coming from three different client processes - which fits the three single caget commands that each ask for one channel.
In the second case you see a single UDP message of length 160, from one client - which fits the case of a single caget asking for three channels. Then, 30ms later, another search request, asking for a single channel - could be that it got two answers by that time and re-sends the missing channel.

Your dumps do not show the name resolution answers (UDP 10.0.6.60 -> 10.4.8.11.35593), where one of the answers causes the error printout.

Note also that caget is a very short lived client. Could be that the client is already exiting when the IOC tries to send back the name resolution answer?!

@jeonghanlee
Copy link
Owner

Yes, I understood exactly what you described. BTW, the trick works well, so I can access three PVs come from three IOCs. It look like we can ignore this error message. Am I right?

If so, I will update the wiki (document) page in new epics site.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ralphlange @jeonghanlee