Disable stacktrace collection per EventPipeSession

[ezsilmar]

Background

Currently, all events emitted from the managed space by EventSource will do a stackwalk, see eventpipeinternal.cpp:121. The obligatory stackwalk leads to significant performance penalty for frequent events while in some cases the stacktrace is not required.

As an example, I'm working on the association between CPU samples from perf record and spans of System.Diagnostics.Activity. The idea is to bring the profiling data on the application level, showing the cpu usage grouped by activity name. To make it work, I record all the activities with Microsoft-Diagnostics-DiagnosticSource and track TraceSyncrhonousWorkStart and TraceOperationStart events from System.Threading.Tasks.TplEventSource to follow the asynchronous flow.

With the stacktraces, the idea is not viable because the application spends more than half of the cpu time doing stackwalks. By disabling the stacktrace recording (just put false in the file linked above), the overhead drops down to around 3% showing really promising results.

Proposed Feature

I propose to introduce the bool collectStacks parameter when creating an EventPipe session. The change is required both in the runtime implementation and the DiagnosticsClient. For managed events the effect of session.collectStacks would be straightforward as there's no way to control it in EventSource definition. For the native we could check session.collectStacks && event.needStack to respect ClrEtwAllMeta.lst

Why I think introducing this parameter on the session level is a good idea:

1. For the same event, the importance of the stacktrace depends on the usecase. Thus the user should be able to choose
2. It seems too complex for me to propagate the flag per provider, even more complex per keyword or per event. On the other hand disabling the stacks with the environment variable (per process) will severely limit other usecases.
3. If we need the events both with and without the stack at the same time, we could start 2 parallel sessions. Not very convenient but not too much trouble either.
4. It's relatively easy to support backward compatibility as we don't change the defaults and the parameter could be optional

Usage Examples

Starting the session:

var session = client.StartEventPipeSession(providers, collectStacks = false);

[tommcdon]

@davmason @noahfalk

[ezsilmar]

I've just realized that RequestRundown could be the thing I need. I'll test it and close the issue if it's the case.

[davmason]

Hi @ezsilmar,

RequestRundown won't have any affect on stack collection. There is not currently a way to disable stack collection and it causes pain for some scenarios, we are in the process of adding options to disable stack collection. PR dotnet/runtime#82470 is about to be merged, and long term I would like to do the approach you describe where we can disable stack collection per session.

[ezsilmar]

Hi @davmason, thanks for the clarification!

There's not much info on rundown events on the internet, as far as I understand it it's basically symbol information?

I was confused with RequestRundown because of these lines:

 if (event.NeedStack() && !session.RundownEnabled() && pStack == NULL)
 {
     EventPipe::WalkManagedStackForCurrentThread(s);
     pStack = &s;
 }

I read them incorrectly at first, assuming that we only collect stacks when we emit Rundown which would make sense to me. However I'm a bit lost as the logic is opposite, meaning probably there are two different ways to do a stackwalk? I couldn't find it in the code.

Anyway, I'm interested in contributing to this subject as I'm going to implement something fast for our custom-built runtime in the coming days.

[davmason]

Rundown is something we do to make sure everything in a trace can be resolved. Jitted methods are dynamically generated at runtime so their IPs are not contained in the symbol file, and we also can potentially have dynamically generated types, in memory modules, and other things that are not known in the symbol file.

When rundown is triggered we go through all the dynamically generated things and emit the proper information so anyone reading the trace can properly map everything. It is effectively iterating through all the method bodies, types, etc and emitting an event for each one.

We turn off stack information for rundown events as an optimization because it's not relevant, there is no reason for anyone to care what stack is emitting rundown events since it is always the same one.

[ezsilmar]

Ah ok I see there's rundown_requested field that is set at session startup but then there's rundown_enabled that is set to true at the end of the session. So Rundown provider doesn't rely on event.need_stack to exclude the stacks but on the separate check in the code.

[ezsilmar]

I've just realized this should be closed as completed as the implementation was merged.