Depending on what you're doing but if its deemed immediately dangerous if the appliance could be used then really it should be capped off by a gas engineer.
In the real world most roofers would take steps to ensure the boiler wouldn't get turned on and that everybody who could turn it on is thoroughly educated in the dangers
I would 100% get someone to check it over before its turned back on as any movement of the flue could break something and be extremely dangerous