Continuity check message (CCM)
What is good to know? (if you are familiar with the basics, go ahead and read)
First of all. CCMs are the heartbeat of the network being monitored. By protocol description, every CCM has few important parameters: Origin (Maintenance domain and association), hello interval and status bits, sequence number and some Organization specific stuff which is irrelevant to this guide. The packet looks like this:
00000000 d4 c3 b2 a1 02 00 04 00 00 00 00 00 00 00 00 00 |Ôò¡............| 00000010 ff ff 00 00 01 00 00 00 96 35 c1 4c 0d 87 05 00 |ÿÿ.......5ÁL....| 00000020 74 00 00 00 74 00 00 00 01 80 c2 00 00 31 00 a0 |t...t.....Â..1. | 00000030 12 4b 2f 30 81 00 c0 03 89 02 20 01 04 46 00 01 |.K/0..À... ..F..| 00000040 3d 88 00 01 04 02 61 31 02 03 6d 61 31 00 00 00 |=.....a1..ma1...| 00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................| 00000080 00 00 00 00 1f 00 0c 00 a0 12 02 2b 44 d3 3f 11 |........ ..+DÓ?.| 00000090 31 8a 00 04 00 01 01 02 00 01 02 00 |1...........|
If you are interested in the details, get the packet and dissect it in Wireshark.
Second. CCMs are transmitted between MEPs. By seeing each other’s CCMs the MEPs establish connectivity. There are however some important prerequisites.
- MEPs in different domains do not connect to each other, no matter what.
- MEPs in different associations do not connect to each other, no matter what.
- MEPs with different hello intervals do not connect to each other, no matter what.
Another important thing. If you have association named MA1 for monitoring VPLS service and association MA2 monitoring VLAN with id 300, you can’t have CFM entities connected between them. No matter the VPLS could label and carry over packets with VLAN tag 300 in Its topology.
Another important thing is the Domain level:
---------- CCM level 7 ---------------------> ---------- CCM level 5 ------> MEP level 5 ---------- CCM level 1 ------X
MEPs filter all lower level CCMs, MEPs process all same level CCMs and MEPs pass untouched all higher level CCMs. This filtering logic is important. If you take a look in the diagram from the previous tutorial page, you will see, that Level 7 domain should not be stopped at any point if we are going to monitor it. And provider or operator domains should not flood CCM packets outside their respective End Points. This logic applies to all kinds of CFM monitored services and VLANs, and It also provides some security and less control traffic over the whole network.
Fourth. You should never forget the MEPs have direction and different behavior when they are facing UP or DOWN (respectively IN or OUT). The diagram below is created for easier understanding when I did OAM CFM presentation for my colleagues in Telco systems. It’s a bit crude, but not hard to grasp. Basically IN and UP MEPs are the same and the words “In” and “Up” have 2 letters. OUT and DOWN MEPs are the same and the words have more than 2 letters. IN/UP MEPs are sending CCM packets in direction towards the other device ports, that are members of the same domain (a.k.a. MIPs) while the DOWN/OUT MEPs are sending CCM packets in the opposite direction. If we visualize the device as a box with 5 holes it will look like this:
Easier to understand MEP direction
The difference in terminology is because of changes that took place while this relatively new monitoring protocol was developed in the last few years and the different vendors that support it. Currently, most (if not all) vendors use the Up and Down naming for MEPs, but if you happen to buy or receive older equipment or older software version – it’s good to know “which witch is which”.
Lot more can be said about CCMs and MEPs, but all in due time. If this article is too basic for you, please look at the OAM CFM connectivity chart.
49 Responses to “CCMs and MEP types”
Sorry, the comment form is closed at this time.
Hi,
In real implementation, how can we specify the direction of packet?ie How can we send the packet towards network only(Down MEP) or how can we send packet towards other port of the bridge(UP MEP)?
Also, when a packet arrived on a port, how do we distinguish that it has come from network side or from the bridge relay entity?
Thanks,
Nithin
If you have a look in the ITU recommendations – there is no such way to mark the CCM frame. It’s not certain if the CCM is originating from Down or Up MEP in the CCM frame – there is no such field in the TLV or the packet header. It’s the role of the code the programmers implemented in the switch to set the CCM (or other OAM) frame’s direction by the information in the switch unit itself. If the MEP originating this CCM frame is with direction Up (IN) then the CCM will be sent by the processor to a port in the same VLAN or service (MIP). If the MEP originating the CCM frame is with direction Out (Down) the CPU will send the CCM frame to the same port the MEP is configured on.
That’s what I’ve seen in my practice with debugging OAM protocols – But surely, every vendor (e.g. Cisco, Extreme, Alcatel, 3Com or my employer Telco systems) can have slightly different implementation.
I hope this answers your question.
Thank you for your proper answer.
Do you know is there any API available in linux to send/recv packet by specifying the direction?
In a Linux PC, if I configure a bridge with two network cards, assuming each port is MEP, Can I
send an ethernet packet from one port to another port(or to N/W) using raw socket API send() specifying the direction?
Also If I get an L2 packet on one of these ports using API recv() , can I recognize the direction of ethernet packet ?
Thanks,
Nithin
You can’t distinguish the MEP direction from the CCM packet alone Nithin. The MEP role is to set the boundary of the OAM maintenance domain. There is one experiment I can suggest for testing your code:
You can simulate raw CCMs with any generator (or your own program if you decide to code it in C as I suspect). But sending from one network card to another will be a bit pointless. Try the following setup:
[SUT] <------> [Linux]Establish link between any port of the SUT (lit. System Under Test) and the eth0 or eth1 of the Linux box. Get this packet and start sending it with a packet generator every second. (I use PackETH, it’s free, small and useful generator for Linux)
Set the SUT’s port with VLAN 3, OAM CFM domain name “a1” on level 1 and Maintenance association named “ma1” monitoring vlan 3 set with MEP id 2 with type OUT/DOWN.
Link the units and see if the connectivity establishes. If you succeed in the packet generation, you will probably see something like this:
If you succeed in this, than your code is good enough. And if you succeed with some magical hack to distinguish the MEP direction from the CCM frames – please, let me know.
Hi,
is it possibile to configure two MEPs having the same VLAN in the same switch?
I explain me better: can I configure 2 DOWN MEPs that belong to the same MA (so have the same VID) on 2 different ports of the same switch?
Yes, for sure.
That’s how CFM based Ethernet protection protocols work. You set a Down MEP on each port that participates in G.8031 (EPS) or G.8032 (R-APS) setup.
The heartbeat packets (CCMs) between the switches are then used to check if the Ring Link is still alive and if not – you switch the traffic over the working link.
Agree with m0rd0r. There is not any field in the CFM packet’s TLV which can holds the direction. It is handled by the programmer by doing like this..if direction is IN/UP then flood the CFM packets to all ports( most of the vendors have hardware to do flooding…programmers have to send CFM packets to a particular port from where hardware does the flooding to all ports ) and if direction is OUT/DOWN then it simply unicast
guys im trying to implement CFM on CIENA but every time I enable CFM it brings my pbt tunnels down, I’m trying for days finding the config issue but cant seem to find it!
Anyone who had this problem before and fixed it??
I am not very experienced with CIENA or Nortel equipment, but can you please explain a bit?
What happens to the PBT and how is it set? MSTP? RSTP?
What tunneling do you use? What encapsulation and how many layers?
Do you filter some MAC addresses? CFM keepalives will not be ruining anything in a working network. They are simple broadcasts to 01:80:c2:00:00:3x
Hi,
Thank you for your detailed introduction, very appreciated.
For one specific MA (in one device):
1. Is it possible to have different MEPs with different direction?
2. Is it possible to allow many up MEPs?
3. Is it possible to allow many down MEPs?
Most of the questions have one answer, and it is “Yes and no – It’s depending on the equipment”.
In most Telco equipment I’ve worked with:
1. In one MA, you can’t have an Up and Down MEP in the same time. Either one Up MEP or several Down MEPs are possible. If you need a setup with Up and Down MEPs – put them in different MAs.
2. “Yes”. But not in the same MA. One MA monitors one VLAN or one Service. You don’t need more than one Up MEP. If there are more – they will use the same MIPs in all cases – you are just putting more configuration, which will actually do the same.
3. Yes. As many as you have ports on your switch/router. They filter CCM’s much better then MIPs, allowing less CPU usage from the Switch/router. They also limit connectivity to other MEPs behind them.
Hi,
I would like to ask you a question about CFM with a situation as below:
Suppose that a switch has 2 ports configured MEPs at different MAs but same MD level
+ Port 1: down MEP, MD level 5, VID 1
+ Port 2: Up MEP, MD level 5, VID2
The question is that when CFM packets with MD level 5 and S-VID 2 come port 1, will port 1 forward these CFM packets to port 2 or discard them because of different VID?
MEP filtering function depends entirely on the software implementation. Cisco does it one way, Extreme does it in more strict way, Alcatel-Lucent don’t know (never used) and Juniper has the less strict MEP filtering.
If you want the CCMs to be forwarded to port 2, port 1 should be a member of both VLAN IDs. Otherwise the hardware will drop the CCMs 100% when it receives them on port 1.
If port 1 is member of both Vlan IDs, you should check if port 1 is also MIP of the other MA (the one monitoring port 2). In this case – it is expected to receive the CCMs targeted for port 2 through port 1.
MEP filtering is very strict only for CCMs of lower levels, it drops them always.
Hi,
Thanks a lot for your explanation. I understood this situation.
I would like to ask you one more question.
802.1ag defines that one MA may have a primary VID and a list of VIDs. Do you think if there is any limitation about the number of VIDs that one MA supports?
Because MD level and VLAN ID of CFM packets arrived is used to look up a MA configured on a switch. If full number of VLAN ID (4096) is supported, it makes the look up table very huge. If hash function is used, the latency of searching may be long.
You cannot control this.
Each MD level should add one “slow protocol” mac address entry in the FDB database per MA like this:
01:80:c2:00:00:MD
MD = 0x30 + MD level.
So, if you have 100 MA’s monitoring 100 VLANs (or services) you will see 100 MAC address entries per monitored port.
Also – MEP filtering will also reserve one MAC address per ALL LOWER MD levels. This way, if you have Level 7 domain with MA’s monitoring 100 vlans on just one port, your FDB database should have 800 entries.
Solution 1: Don’t use MA’s to monitor all cVIDs and sVIDs on your ports. Use 1 MA and monitor just one VLAN with similar path in your network like your other VLAN’s.
Solution 2: If you don’t have really complex topology – Use MD level 0 only. Thus, MEP filtering for lower levels will not need to create more entries, because you use lowest MD level.
Hi,
Thank you for your answer. However, I still cannot catch your idea.
I would like to give more detail information about my question as below
I design a switch which has 12 GE ports and supports a pool of 128 MEPs.
When a CFM packet comes at one port, the switch has to check if this CFM packet belongs to any MA configured on this switch or not. If one MA is matched, this CFM packet is further processed (such as checking MAID, MEPID and updating counters, ..), otherwise it is discarded.
To check MA of this CFM packet, VID and MD level fields of this CFM are used to look up any MA that is being configured in this switch. Currently, I make a look up table with the index being {VID, MD level}. So that the look up table is very huge with 4096 * 8 entries. I am finding a way to optimize this table but still have not found yet.
Do you think if my idea is correct and any better solution for it?
I appreciate your explanation and answers very much.
You are creating the software that will do the actual OAM CFM?
In this case I recommend downloading the standards for dot1ag (http://en.wikipedia.org/wiki/IEEE_802.1ag see at the end) and have a good research how you should react to CCM frames coming on the switch ports.
I will try to answer your question as best as I can – In fact my work is exactly testing such devices against other devices. So if I find any design flows – I will let you know.
Thanks for your help. Actually I am designing a logic circuit to do CFM functions.
Okay, I get it so far.
Please explain why do you need such a big chunk of data for just 128 MEPs? You need to discard all CCMs of lower levels and you only need to react to CCMs that are on the same level and are members of the same VTag as the MA. Why do you need to traverse the huge chunk of data in this case?
Thanks for your reply.
In my switch, I just support 128 MEPs, each of them has one index which is used to access all related information such as time out counter, MEP cross connect state machine, MEP fault Notification Generator State Machine, Remote Error, …
I make a look up table to identify the MEP index based on MD level, VLAN ID and port ID of CCM arriving. If one MEP index is found (enable), this CCM is processed such as compare MAID and MEPID, updating time out counters; otherwise this CCM packet is discarded or forwarded.
For example:
+ I configure a MEP at port 1 with MD level 5, a MA associated with a VLAN that has a VID list {1,100,1000}
+ Then one CCM arriving at port 1 has MD level 5, VID 1 or 100 or 1000, this CCM is processed because one MEP index is found in look up table.
+ If CCM arriving at port 1 has MD level 5, VID 4000, this CCM is discarded because no MEP index is enable in this case.
By using direct access memory, the latency of searching MEP index is low but the large size of look up table is implemented.
Do you have any idea about this implementation?
Still don’t understand why you need so many entries in the lookup table.
128 meps are a simple 2 dimensional array:
MEP ID, MAID (or VTag/STag), MDID, CCM rate, Direction, PORT ID, RDI bit, Fault notification bitmap…
I am a bit rusty at C code, but as I see it – this is less than 4k data in the memory.
And traversing an array of 128 entries should not be slow or hard. The equipment I use has no Limit, you can set 8129 MEPs and they will all generate CCMs at rate of 1 pps. And CFM process takes 10ms to react to a bunch of remote CCMs.
The example below explains why I use the huge look-up table:
+ The Brige supports a MA with MD level 5, VID list ranged from 2 to 4095 (primary VID is 2), list of Remote MEP ranged from 1 – 8191 (means this MA has 8191 remote MEPs). I configure the look-up table that with MD level 5 and VID ranged from 2 to 4095, it indicates this MA enable (by setting bit MA enable).
+ CCMs from remote MEPs of this MA has MD level 5, VLAN ID being one of values ranged from 2 to 4095. VID values from Remote MEP depends on configuring primary VID of Remote MEP which can be one of the VIDs assigned to this MA. It means there is a case that VIDs of CCMs arrived at the Bridge can ranged fully from 2 to 4095.
+ To identify if the CCMs arrived at the Bridge belongs to the MA that the Bridge supports, MD level and VID of the CCMs are used as index to access look-up table, if bit MA enable is set, the CCMs are processed by the Brige, otherwise, it is discarded.
Do you have any other algorithms to check appropriate MA of CCM arriving?
* I am concerning that if an equipment supports full of 8192 MEPs with the highest rage (3.3ms), it can cause the equipment overwhelmed.
Could you tell me how many ports your equipment supports? and how many remote MEPs can be configured for each local MEP and the bandwidth of your equipment?
Well, the smallest switch I use has 4 ports only. 2 for uplinks and 2 for user CPEs.
Largest router device we have is modular and has 4 blades x 64 ports each, but it is not CFM active.
For each MEP, we support up to 8 remote MEPs.
Largest CFM able switch we have is 40 ports, each being able to be DOWN MEP and each being able to address 8 remote MEPs (320 total, with variable CCM rate).
But those are “common sense” limitations and can be enhanced as per customer will.
Thanks for your information. I still have some more questions.
Which is the brand name of the product you are using?
According to your information, I see your equipment just supports one Down MEP per port, isn’t it? Doesn’t it support UP MEP?
And with only 8 Remote MEPs, the Maintenance Domain is monitored by this equipment is quite small.
I am curious about the application of your equipment. Which area (access/metro/core network) is it used?
I work @ Telco systems as a QA engineer and our equipment is quite affordable.
8 remote MEPs are just enough for all the needed operations. Having one down-MEP per port per MA is more than enough too. If you need to monitor one VLAN in the network – why would you need more than one Down MEP on a port for this VLAN?
Down MEPs are used more often at End-points, where you need to end the monitoring of this VLAN or service and do measurements but also ensuring the MEP filtering will stop all other traffic, so no customer equipment hears test packets and seeing you are measuring something in the core network.
It is just common sense.
Using Up MEPs has its other uses and there – having only 8 remote MEPs in connectivity table gives some limitations – yes.
But the software implementation is quite fast, and the workarounds are not that much of a pain.
A good sys-admin knows how to do cluster and divide a network, so it does not take too many domains and MAs for monitoring. It is only a problem when you build R-APS ring with UP MEPs, this may limit you to 8 switches only – but I would not build large rings with UP MEPs because this will generate too much control traffic in the ring and clutter it. I would Use DOWN MEPs instead, which will limit the CCM control traffic only between each 2 switches.
Using Up MEPs in a ring with 8 switches, using 3.3ms CCM rate will generate 2400 packets per second from CCMs only – it is quite a lot. So common sense applies again – use Down MEPs, so you filter the CCMs between each 2 neighboring units only
Thanks for your information. It is very helpful.
However, I am wondering that if only one Down MEP is configured per port, how can I distinguish the responsibility of monitoring one VLAN or service between service provider and operators in some cases.
Do you have some experience with Link Trace message (LTM) and Loop Back message (LBM)?
Do you know in normal cases what the length of LTM and LBM is?
I hope you are not an enemy of the open source 🙂
Download this tool and have a look at the source code. In one of the header files, there is a builder for all the packets you need:
https://noc.sara.nl/nrg/dot1ag-utils/index.html
It does linktrace and loopback (the author calls them eth-ping and eth-trace).
You either snip part of the code you need (it is with new BSD license) or sniff the packets this tool will generate while executing.
I can’t disclose parts of our code, because it is proprietary, but this should help you.
—
Regarding the other question:
If you don’t need (or can’t manage with) just one down MEP per port – code it in any way you like. Just read the standard and make the MEP/MIP filtering function – the way it is supposed to work: Discard all lower level CFM frames, respond to all same-level CFM frames and pass-through all higher level CFM frames.
Stack them in the right order too.
Imagine the following
DOWN MEP |<—-CCMs—–>| Level 1 domain
DOWN MEP |<——–CCMs——————–>| Level 3 domain
If you receive level 1 CCM and you have level 1 and level 3 down MEPs on a same port – what will happen?
If the DOWN mep of level 3 simply discards the CCM frame, level 1 domain will not have connectivity.
So you need to FIRST complete the logic for level 1 DOWN MEP and then remove the CCM frame from the stack/tail/list/whatever, and THEN execute the logic for the level 3 DOWN MEP.
HTH.
Thanks a lot for your help. I greatly appreciate your kindness. Have a nice week!
how many MEP can create in a single switch?
By definition, the limit is 8192.
But I don’t think you need that many. And if you are using fast CCM generation like 1 CCM every 3.3 ms – you will turn your switch into a traffic generator (8192*300 = 2457600 packets each second).
Hi,
Wondering what would be the use case for MEP associated with per CE-VLAN COS field? Though i don’t see this mentioned in 802.1ag/Y.1731, MEF17 (section 8.7) talks about ME with different priorities. Won’t this make MEP filtering rules in a switch more complicated?
Not at all. Class of service SHOULD NOT affect the CCM generation and filtering.
There are few things that can break CFM connectivity:
Filtering per MAC address (multicast destinations in particular).
Filtering per VLAN.
Filtering per Priority (VPT).
Rewriting VLAN header of the frame with another VLAN ID.
Traffic shaping in general must be set to not filter management traffic (e.g. – do not apply shaping policy to target with multicast MAC address destination).
If you start filtering/shaping management traffic, most routing/switching protocol will also fail – not just CFM.
MEP filtering function is based on the last octet of the multicast address and the MD/MA/MEP ID inside the packet. If some CoS policy changes them (I don’t know of such QOS rule at all) – the MEP filtering will most definitely fail.
Hi,
I know that a MEP can send multicast CCM packets but I don’t understand a thing:
a DOWN MEP may receive CCMs from different remote MEPs that belong to the same MD-MA?
Yes. Imagine your DOWN MEP is connected to a remote unit’s MIP port and you have 3-4 UP MEPs on this unit (or other adjacent units). Your MEP must establish connectivity with all remote MEPs who send CCMs.
I have created an UP-MEP on an operation down port but when I checked the counter Tx value is incrementing but Rx value is zero.why??
Do you have a remote switch configured with similar configuration?
Connectivity between MEPs needs a pair of MEPs with unique IDs within the same domain and association. Else – there will be no connection.
If you already have remote MEP, check if it is in the same VLAN/Service, check if it is enabled to send CCM, check if there is something between the 2 MEPs that is blocking their communication.
If there are other switches in between, check if the MIP creation policy is enabled and there is no filtering.
Better to start this with a back-to-back topology and then build up.
I’ve a question about a race condition with using MEP’s. I’m using Telco Systems MPLS gear for many years now and I’m trying to set up a CFM span between a Juniper router and a 7224 in another city.
The issue I have is that I’ve got everything working so that CCM’s flow and when a port goes down in one location the corresponding port at the other end goes down as well using action profiles. But since each port goes into shutdown state, they will never come back up unless I disable the action profile at one end.
I’m trying to emulate a line state pass-thru, so what is happening is port A goes link state down, CCM fails and action causes Port B to go into shutdown. Well since CCM has failed now from Port B, port A goes shutdown as well. Then even if you plug the cable back in to port A to bring the link back up, since the port is in shutdown it won’t restore.
I don’t really understand how I can resolve this since that’s part of the design of a Level 7 CCM service.
See if your TMetro7224 supports Event Propagation profiling.
There is a rule to get port UP if MEP is back UP. Not sure if 7224 supports this though.
Ask Telco’s support by email if it doesn’t, they are responsive 24/7. And can probably pop-up a solution for your case.
There is event propagation and that’s what I’m using to take the port down when the MEP goes down. The issue is if the port goes down on each end of the circuit, there is no way for the MEP to ever return to the UP state.
Since I’m trying to emulate a link state across a network, when one port goes down, the other follows. So if I have a provider that reboots their router connected to me, the link state on the port goes down (but admin up), then that triggers the MEP to fail, and then event propagation on the other end shuts the port down. When that port goes shutdown, then the MEP on the original end detects the CCM failure and shuts it’s port down.
Now I have 2 ports both in shutdown and once the customer’s router reboots and tries to link up, it can’t because the port is in physical shutdown.
That seems to be the issue, protecting the transport is fine, but I can’t seem to protect at a port level.
Not sure if this is solvable in your case. Event propagation has a reversive action and will recover the port state if the remote MEP goes Up.
Unfortunately, I am not sure how it is done with Juniper exactly. Check if their documentation has some hints.
Just to close the loop, this is not solvable in my case, specifically the Telco equipment. The Juniper has a feature that sends extended CCM’s with a status of interface down due to lower layer or admin, Telco doesn’t support this.
So unfortunately, if one port goes down, the other will be taken down due to CCM loss and at that point you are in a dead lock and neither will ever come back up without manual intervention.
Sorry.
I think newer Telco devices got this solved. 7224 is the grandpa of Telco routers and is not the best in line.
Hi,
I have few question on Ethernet CFM.
1. Can a switch have two Down MEP with remote MEP as each other, e.g. MEP say 100 has its remote MEP 200 and MEP 200 has remote MEP 100 and both MEP 100 and 200 lies on same switch ( same MD Level and MA).
2. What happen when CCM is disable (i.e CCM-enable is false) and the MEP receive a CCM packet for its MD Level and MA?
Thanks
Neelabh
1. Yes, definitely. But drawing a diagram and seeing repeating MEP IDs is a bit confusing.
2. It’s discarded if the protocol is up. It goes through if the switch doesn’t support CFM and the port is in the same VLAN.
Hi,
need to define number of up/down MEPs on the switch that will support Y.1731. So, 2 up mep per EVC for CC, etc. Do I need 2 down MEPs per service (not per port) for some reason? You mentioned service check without showing to User equipment that something is done on the network (not quite clear why user eq will see CC frames coming to up mep, but anyway)…any other reasons? except ERP protection, etc.
Thanks
Andrew
I don’t think I understand the question. What is the assignment?
Hi M0rd0r,
If my CFM network has standard nesting of customer, provider and operator domains, can i create an all up-mep or an all down-mep configuration for the complete network? Any documentation would also really help.
Thanks
MEPs should face the segment of the network they monitor. Up MEPs should face the core, Down MEPs should face the customer. It is a bit confusing at the beginning. For examples, I need to at least know what equipment you use and if I am familiar with it.