Enough validation?

[cubakija]

I am relatively new in DES.
I've done schedule based simulation, which objective was to find max yearly equipment throughput.
Orders are generated by schedules which use empirical distributions. This distributions are calculated based on historical data of one year.
Total throughput consist of sum of throughputs of N different products.

I validate model using face only face validation, as I don't know what statistical validation would be appropriate in this case. I have only one set of historical data, which I already used to find empirical distributions.
Then I make an experiment which consist in increasing number of arrivals till the throughput stops increasing. Next, I analyze model using 95% confidence interval of total throughput as a performance measure.

My question is: Whether I follow correct procedure, and whether the face validation was good enough in this case?

Thanks in advance.

[kpeacock]

Although face validation is quick cut and dry method to determining the validity it maybe in your best interested to try at least two other methods for validating your model. Two methods that I think could be of interest to you would be to use the historial data. This way you compare to see if the model is a true representation of the environment. The second method would be to used fixed values first on items in the model and through some simple calculations you can see that your model is running correctly.

Good Luck

[ewilliams]

Two excellent ideas from KPeacock. Also, show your simulation results to your client. It is vital that the client consider them reasonable -- that earns your model validity AND credibility. One validation test, based on this idea, is called the Turing test. How it works (assuming an existing system is being simulated): show the client real system performance metrics (e.g., from last week) and simulation results, but do not say which is which. If the client cannot tell which is which, your model passes the Turing test. If the client can, your model fails the Turing test. In this case, you ask the client "How could you tell them apart?" The client might say, for example "The real system never has a long queue in front of machine X, so the result with the long queue there must be the model one." Now you have a valuable lead for correcting and improving the model.

[cubakija]

tpeackok, ewilliams, thanks for the valuable suggestions.

I've done this model for a company where I am working as process engineer, so in this case analyst = client.
From the client side of view, I can say that model passed Turing test.

I was thinking to use a historic data to get the average year capacity. Then I could use t statistical test to check a null hypothesis (there is a difference between the model and system).

I guess that with the both test passed I have enough validation.

Best regards

[ewilliams]

Congratulations on passing your Turing test! That is always a major step forward. Your next step is good: drive the model with last year's data and use a t-test across a few (5 or 10?) replications to see if performance metrics are statistically indistinguishable from the metrics observed during the last year. If they are (e.g., you are hoping to accept the null hypothesis that "mean simulation performance metric prediction = last year's actual value") then drive the model with various predictions of next week's/month's/year's data and use the corresponding performance metric predictions to guide managements' decisions.