20. Notifications and Alerts
ChiefNET monitors for a number of possible abnormal conditions and raises events. The events cover a number of areas of the ChiefNET device.
An event is either
If you are a customer administrator, you can set up the alerts and the thresholds. These alerts are for you to monitor the system, and are not always indicative of a failure. The messages indicate whether a problem was detected. For some problems the ChiefNET's device or the orchestrator automatically takes a failover approach and there is nothing for you to be done. For some other problems, the message indicates whether the administrator should review.
For each alert, you can set to receive the notifications by email or just stay on the webui. All alerts, no matter how they are delivered, are available on the WebUI and are kept upto 30 days. Alerts older than 30 days are deleted automatically.
20.1 Viewing Alerts on WebUI
Once you log in as a customer administrator, the alerts can be seen by clicking the bell icon on the top right corner section of the WebUI. A new section will open up on the right side of the WebUI screen.
Scroll down on the right-side section and click on 'View more' for details and advanced filtering. This will show a table of all the alerts, with their timestamps, the alert type, the device where the alert originated and a description.
Filters
Filtering may be done based on the kind of the alert.
- All - all notifications are displayed
- Alerts - Only those that are of concern are displayed under this filter.
- Device Configurations - All configuration changes are logged and displayed under this filter.
- Device Provision - If new devices are added to the netweork, they are displayed under this filter.
- User Activities - All user login/logout events are logged under this filter.
For each filter-type displayed as a horizontal bar, the time duration may be selected
- Today
- Yesterday,
- Last 7 days
- Last 30 days
- This calendar month
- Custom Range
The red-colored superscripts indicate the total number of events under that category.
Additional Tips
The table can be exported as a CSV file to help with reviewing, troubleshooting or auditing purposes.
Additional actions such as column-level filtering, whole table search, deleting a specific event are all available. These features are similar to those found in other tables of the ChiefNET WebUI.
20.2 Notification by Email, Telegram
On the left pane, choose 'Settings' and click on 'Notification' tab to manage how you want to get notified when events occur. You can also set the thresholds for some notifications to identify conditions that would be of concern.
The default thresholds and the notification settings are apt for almost all customer deployments. Below is a description of those notifications, to help with customization, if needed.
20.3 Network Events - Summary
The following table summarizes the list of all events related to the networking aspects of ChiefNET, LAN/WAN ports and links and the network quality.
|
Alert: WAN Interface Bandwidth Utilization
|
WAN Interface BW > 70%
|
-
Raised on a WAN interface when the RX bandwidth exceeds the configured percentage of the total bandwidth. The total bandwidth or the maximum expected bandwidth is configured as part of the WAN interface.
-
This alarm should be enabled only when the ISP link has a known maximum bandwidth. Many ISPs provide a nominal maximum bandwidth but allow more than that in reality. In those cases, the BW utilization is calcualted against a user-configured expected max bandwidth value.
-
Auto-Cleared when the bandwidth utilization goes below the threshold.
|
|
Alert: WAN Port Status
|
A WAN Port is operationally down (i.e. the physical link is down)
|
-
Raised when a WAN port is operationally down. Occurs when typically a cable is disconnected or an adjacent equipment fails or powered down.
-
To clear the event, check if the WAN port is connected to the Internet-side equipment properly. Check the cable quality and the power status of the adjacent equipment.
-
|
|
Alert: WAN Link Quality
|
System detected packet loss on the WAN links
|
-
Raised when the system detects packet loss on the WAN links. A link is considered unstable if the packet loss percentage is > 11%. A link is considered unreachable or unusable when the packet loss is > 90%.
-
This condition is automatically cleared when the WAN link becomes operationally up. Typically, a WAN link goes down due to outages on the Internet Service Provider site. It is almost never an issue with ChiefNET device itself.
|
|
Alert: WAN IP Change
|
IP Address of a WAN Port Changed
|
-
Raised when the system detects that a WAN port's IP address is changed by the ISP.
-
This is an informational event. There is no clearing action. Typically a WAN port's IP address does not change. However, we have noticed that the ISP sometimes changes the WAN circuits causing the IP address of ChiefNET's WAN port to change. When this occurs, even though the WAN port may operationally stay up, a change in IP address makes the tunnel connections to be restarted. This will also cause ChiefNET management connection to be restarted.
-
This is typically a result of the ISP's outage issues on their access network. It may also occur if ChiefNET is connected to a firewall of the customer, and if the customer changes the configuration on the firewall.
-
ChiefNET automatically handles this change of IP event and restarts the tunnels as necessary. No operator intervention is required.
|
|
Alert: WAN Failover
|
Traffic switched automatically from a non-working port to a working port as configured
|
-
Raised when the traffic is switched from one WAN port to another as configured.
-
This is an informational event. There is no clearing action. There is no need for operator intervention.
-
ChiefNET continuously monitors the WAN port status and runs through the configured policies. When a WAN port fails, traffic may be switched to a working WAN port. This alert is raised when such a condition occurs.
-
There is no need for operator intervention. This may be used by the customer to identify unstable WAN provider links. This issue is almost never due to ChiefNET itself.
-
Usually this event is associated with the 'WAN port status: down' event. It is a useful alert for troubleshooting.
|
|
Alert: VPN Status
|
a VPN/SDWAN Tunnel Link is down.
|
-
Raised when a VPN/SDWAN tunnel is down.
-
If the VPN/SDWAN tunnel is down, it is automatically retried. If it stays down for more than a minute, it is indicative of a WAN Internet service provider problem or the WAN link quality issue.
-
Operator intervention may be required when the VPN link stays down for more than five minutes.
|
|
Alert: VPN Link Quality
|
VPN Link quality is poor due to higher than expected packet loss.
|
-
Raised when ChiefNET observes packet loss on the VPN link.
-
This is an informational event. There is no clearing action. There is usually no need for operator intervention as the conditions may just be temporary.
-
This event is automatically cleared when the WAN link (ISP provider links/devices) becomes stable.
-
The VPN link is considered unstable if the packet loss is > 11% and considered unreachable/unusable when the packet loss is > 90%.
|
|
Alert: Tunnel WAN Change
|
A VPN/SDWAN tunnel changed from one WAN to another as part of a failover
|
-
Raised when a tunnel is found to have switched over to another WAN port
-
This is an informational event. There is no clearing action. No operator intervention is required.
-
This event occurs as part of the WAN port failover event. This reassures that the tunnel is indeed using another WAN port as part of the configuration.
|
|
Alert: Tunnel Latency
|
Packet latency on a tunnel is > 3s
|
-
Raised when the ChiefNET observes the probe packets on a tunnel experience a latency of more than 3 seconds.
-
This usually occurs when the WAN link is unstable and is usually temporary. If this persists for a long time, operator intervention may be required to find out the WAN link that is experiencing the outage. This again is almost never an issue with ChiefNET itself and is due to the ISP network outages and the poor ISP link quality.
|
|
Alert: LAN Port Status
|
A LAN port is operationally down (i.e. the physical link is down)
|
-
Raised when a LAN port is operationally down. Occurs when typically a cable is disconnected or an adjacent equipment fails or powered down.
-
Cleared when the LAN port is operationally up, either by operator intervention, or automatically.
-
If the LAN Port is not connected to any network, change the administrative status of the port to the DOWN configuration. The port will be down, but the alert will not be raised.
-
If the LAN Port is supposed to be in use, verify the cable connection and the adjacent equipment. Any power failure or a poor cable can cause the LAN port to go down.
|
|
Alert: Ping Latency
|
Ping packet latency > 3 seconds
|
-
Raised when the ping latency to a configured destination is more than 3 seconds.
-
This is usually a troubleshooting tool to identify poor WAN link quality or ISP network outages. This condition usually resolves automatically within a few minutes as the ISPs reroute their network traffic. However, if it persists for more than a minute, the operator must verify the ISP network quality.
-
Usually when this issue occurs, it is a result of poor WAN network link and a configured traffic policy causes the traffic to switchover to a higher quality WAN link. Hence the impact to the customer traffic is minimal.
|
20.4 Security Events - Summary
The following table summarizes the events related to the overall access controls and management security of ChiefNET.
|
Alert: User Activity
|
User login/logout/registration events
|
-
Raised when a user logs into the Orchestrator WebUI, logs out or when there is are similar user account activities.
-
This is an informational event. There is no clearing action. The operator should ensure that these are authorized users that use the ChiefNET Orchestrator.
|
20.5 System Events - Summary
The following table summarizes the system-level events related to resources within the physical/virtual CPE device itself, such as the CPU/memory/disk resources.
|
Alert: System Power
|
System power outage was detected
|
-
Raised when the ChiefNET devices comes back up from a power outage.
-
This is an informational event. There is no clearing action. This indicates that the power to the ChiefNET devices experienced an outage. The operator must ensure that ChiefNET device is continuously powered. In some locations, the use of UPS will reduce the unexpected outages.
|
|
Alert: Memory Usage
|
System Memory usage is > 70%
|
-
Raised when the system uses its main memory beyond 70% of capacity.
-
This is an informational event. There is no clearing action. Operator intervention is not necessary. This is monitored by ChiefNET service team for all devices to identify anamolies.
|
|
Alert: CPU Temperature on a Physical CPE
|
Temperature > 85 deg C
|
-
Raised when the CPU temperature rises above 85 deg C
-
Auto-Cleared when the CPU temperature is found to at or below the threshold.
|
|
Alert: Device configuration
|
n/a
|
-
Raised when the device configuration has been changed by a user logged into ChiefNET WebUI.
-
This is an informational event. There is no clearing action. See below for actions if this is an unexpected configuration change.
|
|
Alert: Device provision
|
n/a
|
-
Raised when the device configuration has been changed by a user logged into ChiefNET WebUI.
-
This is an informational event. There is no clearing action. See below for actions if this is an unexpected configuration change.
|
|
Alert: Disk Usage
|
Current disk usage > 90%
|
-
Raised when the the disk space usage on the physical CPE exceeds 90% of the total.
-
Cleared when the disk usage falls below the configured threshold.
|