C1W2 2.5 Output not in /opt/ml/processing Dir

I was reviewing the logs in CloudWatch. Per log message, different output was written to a directory. One example is a PDF as shown here:

INFO:analyzer.report:PDF report ‘/opt/ml/processing/output/report.pdf’ generated successfully.

I went into the system terminal for my instance and was unable to locate directory /opt/ml/processing.

Any suggestions on how to access the output?

I see that the output is in the directory /bias/unbalanced/output/

1 Like

Hi @community_token, I think you already got the answer. Yes, you are right all outputs are in the path of S3 which we specified in clarify.DataConfig(). in this case ‘bias/generated_bias_report/unbalanced’.

Additionally, the reason why you are not able to access the path of output described in the CloudWatch with your system terminal, in this case ‘/opt/ml/processing’. This is not the path of JupyterServer instance. Instead, this is the path of another EC2 instance in which our processing job is actually running. Please refer to the image below even if this is not the architecture of the Processing job, however, you might understand its base mechanism.

Happy learning!


source: Amazon SageMaker Debugger - Deep Dive

1 Like