When your outlier filter is active, an email with a reply time higher than the number of hours set in the outlier filter, will be excluded from the calculation of your average reply times.
The outliers filter can be found under Tools > Filters
To turn off your outliers filter, click on the blue tick/checkmark on the right of the outlier filter. You’ll see your reply time statistics recalculate when you turn the Outlier filter on or off.