TON Status

Mainnet Validators and Liteserver Owners

Please update your node software (see "Target versions"):

update
upgrade master


Target versions:
— mytonctrl: 33bd174
— node ea0dc16
If you are not using mytonctrl, check this instruction.

This update is mandatory for validators and liteservers. Node changelog. Mytonctrl changelog.

If you have several validator nodes, please update them one by one (update, wait for synchronization, move to the next one).

Mainnet validators and liteserver owners
Scheduled network update on December 17

We are asking validators to schedule a time on December 17 at 11:00 UTC for validator software update.

This update is mandatory and, beside other things, improves work with db, adds convenient option for key backups, removes duplicative calculations by caching, introduces new opcode in tvm and more.

Mainnet Validators and Liteserver Owners

Please update your node software (see "Target versions"):
update
upgrade master


Target versions:
— mytonctrl: e0ead70
— node eed3153

If you are not using mytonctrl, check this instruction.

This update is mandatory for validators and liteservers. Node changelog. Mytonctrl changelog.

If you have several validator nodes, please update them one by one (update, wait for synchronization, move to the next one).

Mainnet validators and liteserver owners
Scheduled network update on October 28

We are asking validators to schedule a time on October 28 at 10:00 UTC for validator software update.

This update is mandatory and, beside other things, decreases network consumption, and substantially improves synchronization and garbage collection speed.

Dear TON Validator

We would like to notify you that from the 26th of September 10:00 UTC we are expecting an increased load on the TON blockchain, as the Hamster Kombat game project with more than 100 million monthly active users will be minting coins on the blockchain, which is a unique and first event of this scale for the blockchain industry.

We kindly request from September 26 to September 29:

1) be in touch. Follow the @tonstatus channel. If emergency actions are required apply within the hour.

2) constantly monitor the status of your validator and hardware during these days

The quality of validators directly affects the quality of the blockchain. We appreciate your participation in The Open Network.

Found a bug in node that caused performance issues overnight and today. Some validators have been updated, which solved the main problem.

Block production has been restored in all shards. It will take some time to process the accumulated messages (~ 1 hour), during this time there may be a slight performance degradation.

Upd: https://telegra.ph/Report-on-September-13-2024-Operation-Incident-09-14

The network is experiencing performance issues. Transactions may take longer than usual to complete. A fix is being worked on.

Block production in the shard has recovered. The situation is currently under review.

Block production in the shard 0x9000000000000000 has stopped. The other shards are working normally. The core team is investigating the problem.

Mainnet Validators

Please update your node software (see "Target versions"):
update
upgrade master


Target versions:
— mytonctrl: 74e536b a467af5 (updated)
— node: 1bef6df

If you are not using mytonctrl, check this instruction.

This update is mandatory for validators. Node changelog. Mytonctrl changelog.

After upgrade please remove from /etc/systemd/system/validator.service all recently added flags (they are not required anymore):
-F *:*:*,
--catchain-max-block-delay 0.5
--state-ttl * (note, if you intentionally set specific state-ttl, keep it as you need)


If you have several validator nodes, please update them one by one (update, wait for synchronization, move to the next one).

P.S. We encourage validators to subscribe to @tonstatus_notifications channel with notifications of penalties for poorly performing TON validators.

Mainnet validators
Scheduled network update on September 11

We are asking validators to schedule a time on September 11 at 9:00 UTC for validator software update.

This update is mandatory and introduces catchain, serialization, network and collator configuration updates that optimize work of validators.

During preparation of this update we focused on minimal and safest changes which can be reliably tested in short period of time from one hand and substantially improve stability of validation from the other.

APIs for validation and effectiveness of validators

1) https://elections.toncenter.com/docs - use this API to get information about current and past validation rounds (cycles) - time of rounds, which validators participated in them, their stakes, etc.

Information on current and past elections (for the validation round) is also available.

2) https://toncenter.com/api/qos/index.html#/ - use this API to get information on the efficiency of validators over time.

This API analyses the information from the catchain and builds an estimate of the validator's efficiency. This API does not use the checkloadall utility, but is its alternative.

Unlike checkloadall which works only on validation rounds, in this API you can set any time interval to analyse the validator's efficiency.

How to use:

- pass ADNL address of your validator and time inverval (from_ts, to_ts) to API. For accurate result it makes sense to take a sufficient interval, for example from 18 hours ago the current moment.

- get the result. If your efficiency percentage field is less than 80%, your validator is not working properly.

- It is important that your validator participates in validation and has the same ADNL address throughout the specified time period.

For example, if a validator participates in validation every second round - then you need to specify only those intervals when he participated in validation. Otherwise, you will get an incorrect underestimate.

- this works not only for masterchain validators (with index < 100) but also for other validators (with index > 100).

Recommendations

1) Please check the efficiency of your validator and in case of low efficiency - take action to fix the problem. Contact technical support @mytonctrl_help_bot if necessary.

2) Please set up dashboards to monitor your validators using these APIs.

Tool for assessing the effectiveness of validators

The new version of mytonctrl has added a new command check_ef which outputs your validator efficiency data for the last round and for current round. This command retrieves data by calling checkloadall utility.

Previous efficiency score from the status command outdated and has been removed.

To update mytonctrl, type update in the mytonctrl console.

Then type check_ef in the mytonctrl console.

Note that the previous round may not be displayed immediately after the update.

Note that the current round data becomes more accurate towards the end of the round.

Validators with an index less than 100 (sorted by effective stake) please ensure that your efficiency is greater than 90% (for the full round period).

Validators with an index greater than 100 will not receive penalties next week as they do not participate in masterchain validation, but please follow the validator guidelines, as you will be included in the penalty system in future updates.

If you need tech support please contract @mytonctrl_help_bot (validators only).

ANNOUNCEMENT: DECENTRALISED SYSTEM OF PENALTIES FOR POORLY PERFORMING VALIDATORS

The current system of penalising poorly performing validators will be fully operational next Monday, September 9.

How do validators determine that another validator has performed poorly?

The TON is supplied with the lite-client utility. In lite-client there is a checkloadall command.

This command analyses how many blocks the validator should have processed, and how many it actually processed in a given period of time.

If the validator processed less than 90% of the expected number of blocks during a validation round, it is considered to be performing poorly and should be penalised.

Technical description of the process: https://github.com/ton-blockchain/TIPs/issues/13#issuecomment-786627474

When and by whom a complaint is filed?

After each validation round (~18 hours), the validator stakes of validators that participated in that round are on the Elector smart contract for another ~9 hours.

During this time, anyone can send a complaint against a validator who performed poorly in said round. This happens onchain on the Elector smart contract.

All validators do not need to send a complaint.

How a complaint is validated?

After each validation round, validators receive a list of complaints from the Elector smart contract and double-check them by calling checkloadall.

If a complaint is validated, they onchain vote in favour of that complaint.

These actions are built into mytonctrl and happen automatically.

If the complaint has 66% of the validators' votes (by their weight), a penalty is taken off from the validator's stake.

What is the size of the fine?

The amount of the fine is fixed and equals 101 TON, which is roughly equal to the validator's income per round.

Where does this fine go?

The fine is distributed among the validators minus network costs and a small reward (~8 TON) to the validator who sent the correct complaint.

When was this functionality made?

This functionality was made in mytonctrl back in February 2021 https://github.com/ton-blockchain/TIPs/issues/13.

The complaints and fines functionality in the Elector system smart contract was made initially at the time of network launch.

Why didn't the penalty functionality work well earlier?

The network fee for sending complaints was significant, which as the Toncoin exchange rate increased, made it uneconomical to send them.

This has now been fixed by optimising the complaint message.

In a week from 9 September 2024, an automatic complaint sender will start working on several nodes in the network.

Is the system decentralised?

Yes, anyone can send a complaint, the penalty is only applied by a quorum of validators on the network.

There is no way for anyone to single-handedly fine anyone.

How do I prepare for the start of the penalty system?

Since this functionality is already implemented in node, you don't need to do anything to make the system work.

Please make sure you're complying with the validator guidelines.

During the week, we will publish additional tools and best practices for monitoring and maintaining the effectiveness of your validator.

Will the penalty system get stricter in the future?

Yes, the audience and the number of transactions in TON is growing rapidly and it is vital that the quality of work is at its best.

The system will improve and fines will increase this year. All updates will be announced in advance.

It makes sense to set up hardware, monitoring and validator work properly. If you don't want to do this please consider using staking services https://ton.org/stake.

Important Instructions for TON Validators

Dear TON Validators, please follow these crucial steps to ensure network stability this week and to avoid new slashing penalties in the future.

Essential Actions:

1) Verify you're on the stable_testnet branch (commit 97c57c3). If not, update: https://t.me/tonstatus/130. For multiple validators, update one at a time.

2) Ensure your hardware meets or exceeds system requirements: https://t.me/tonstatus/102. Upgrade if necessary, one at a time.

3) We imperatively request you to use mytonctrl.

In your mytonctrl console:
- Update to the latest version: update master
- Enable telemetry: set sendTelemetry true

4) Set up monitoring dashboards for RAM, Disk, Network, and CPU usage. For technical assistance, contact @mytonctrl_help_bot.

DOGS Project Alert:

This week, the DOGS gaming project (50 million active users) is minting and listing, causing increased network load. We experienced two network outages on August 27 and 28. Please:

1) Stay available this week and respond within 1 hour, around the clock.

2) Closely monitor your hardware. Contact @mytonctrl_help_bot immediately if you need help.

3) Follow @tonstatus and be ready to apply urgent updates if necessary.

‼️ Updates on Slashing Mechanics for TON Validators:

The TON Core team is working on implementing of new slashing penalties for non-performing validators. Executing the actions above is crucial to mitigate the risk of losing rewards for validating the network.

Your cooperation is vital for maintaining network stability and TON's prosperity. Thank you for your prompt attention to these matters.

URGENT - Mainnet validators

Please make sure you have done the previous update. If you haven't please do.

Then update node flags:

1. open /etc/systemd/system/validator.service
2. remove -F 39987437:600844:7 -F 39987437:600845:7 if they are still present
3. add flags -F 39991868:601006:7 --catchain-max-block-delay 0.5 to the end of ExecStart
4. restart validator:

systemctl daemon-reload
systemctl restart validator

URGENT - Mainnet validators

Please update your nodes and then restart.

In mytonctrl run:
upgrade stable_testnet

If you don't use mytonctrl manually switch node to stable_testnet branch.

If you have multiple validators you can update them all at once. Please stay tuned for a few hours, in case of subsequent instructions.

Mainnet validators

Block production is restored and stable now. Thank you for cooperation!

For now you can delete -F 39987437:600844:7 -F 39987437:600845:7 from ExecStart when it will be convenient for you. After that please run:
systemctl daemon-reload

Note: there is NO need to restart validators after removing flags.

In coming days, we will came with update for the node to mitigate recently discovered issues. Please stay tuned.