A. Intro
This post came about due to a disagreement on a thread about the PCC vs. Chiv. Chain. We all had slightly different opinions on the subject. I honestly don't care which one people choose to wear, if I could afford a PCC I'd probably wear it just to look cool^^. However, it became apparent that some people are not using a valid method of parsing data. Faulty methods lead to faulty data and therefore incorrect conclusions. It's just a game, and in the grand scheme of things not too important. Nonetheless, I thought I'd try to improve things.
I could have simply posted my method said, "Trust me." Instead, I will try to persuade anyone willing to read this somewhat longish post. Do not expect a Master's Thesis on the subject. I understand that there is a wide range of age and education levels in this community, so the post will be kept somewhat simplistic.
B. The Scientific Principles
1. Skepticism: Do not accept conventional wisdom. Anything not proven by this method is only opinion. ("The Scientific Method" is the accepted norm of scientific inquiry, and has been for hundreds of years.)
2. Objectivity: Simply put, be honest and leave your ego at the door. The right answer will benefit everyone, including yourself.
3. Reproducibility: Others must be able to perform the same test and get the same results.
4. Transparency: You must disclose everything you did and how you did it, down to minute detail. This goes hand in hand with Reproducibility.
C. The Method
1. Control groups.
When you begin an investigation, you must have some hypothesis (unproven assertion, educated guess) that you want to prove. In generic terms, this can be expressed as X=!Y (=! meaning "not equal"). There must be a standard (ie, the conventional wisdom) that perhaps you think is not as good as an alternative. For this group, you will obtain data, but you will not truly experiment on this group. For the control group, you will simply observe it, and then compare the experimental group(s) to the control group.
2. Experimental Group(s)
After observing the control group, you will now begin the experimental trials. One variable, one thing and only one thing, will be changed from the control group, and you will observe and record the results. It is important that only one variable be altered at a time, otherwise you will have "confounding variables", ie. you will be observing not only the change from the control group, but also the interaction of the two variables.
In real life, and even in this game, some things are beyond our control. When this happens you end up with multiple variables, which add "error" to your data collection. This does not invalidate your study, as long as you recognize the error and can minimize or at least quantify the error. There is more on this later on.
3. Measurements
3a. Standard Units: When you record your data, you will need to do it in a format that others understand and can duplicate. In a sense, measurements are another comparison. There must be a clear, uniform and unchanging scale of measurements so that the relative values of each data point can be assessed. FF11 already provides a highly uniform scale for the effect of our actions, it is "Damage". However, this is not enough by itself.
3b. Damage/Time: That damage occurs over time, and damage without an accurate measurement of the time over which it occurs is meaningless. Consider this, you and your friend are in line at the bank to deposit paychecks. You ask him how much his check is worth, and he says $5000. Your check is only worth $3000, so you think, "That lucky SOB, heh." However, you didn't ask how long it took him to earn $5k. You get paid twice a month, but he gets paid only monthly, so in actuality you earn more money. But since you didn't ask, you don't know that and you came to the wrong conclusion. Time is another highly uniform scale of measurement, and with highly accurate digital timers readily available, everyone has the ability to measure it. Any event that occurs over a period of time must be normalized for that time (must have that time accurately measured and factored into the data). Almost all events occur over a period of time, even if that time interval is very small. Therefore, Damage/Time is the only proper method of expressing melee damage.
3c. Damage/Mob: I only mention this to refute it. Every data point has at least two coordinates (x,y). Thankfully, FF11 is not as complicated as RL, and we won't have to deal with 3 or more coordinates. For the purposes of measuring your effectiveness as a melee DDer, the x-coordinate is always damage. When you set up your method of measurements, you determine what your second coordinate will be, even if you don't realize it at the time. When you measure damage per mob killed instead of per unit of time, your y-coordinate is the mob's HP.
Allow me to explain. Events that occur over time, like melee damage, have a starting and a stopping point. You control the starting point by choosing when to engage the mob. The end point occurs when the mob dies. If you are carefully measuring time, then that end point can me expressed as a unit of time. But if your end point is only when the mob dies, untimed, then your end point is simply the exhaustion of it's HPs. If you use this method and are solo, then your Damage/Mob HP ratio is always equal to 1 (unless you lose the fight or run away). Obviously, if the value of all your data points equals 1, then you haven't collected any useful information.
If you are in in a party with other DD'ers, then the end point is (Mob's HP)-(Damage Done by Others). In this case you are competing with the other party members for damage, and you are actually comparing (measuring) your damage against theirs. Now, we all know that different mobs, even of the same type have different HP totals. Also, your fellow party members will not output damage at a constant rate; they will have damage spikes and damage troughs (lows) that will change from mob to mob. So, you are measuring damage against a non-standardized, constantly shifting "scale." Also, your test is not reproducible, since what are the odds that someone else will get all the other DD'ers in your party together again without any of them having any new merits/gear/etc? Time, however, is highly standardized and is always available for measurement.
3d. Damage per Session: This has roughly the same down-fall as Damage per mob. Your y-coordinate is (Sum of all Mob's HP)-(Sum of Other DD'ers damage), and you are probably using this method because you don't know how many mobs you fought or how long the party lasted. Again, you are measuring yourself against the shifting, non-standardized measurement of your party members.
3e. Bi-modal data points: These are events that can be expressed as simple yes/no results. These do not need to be measured over time (but can be if desired). Examples of these would be Hit vs. Miss, Critical Hit vs. Normal Hit, etc. Spell damage could (for some purposes) be measured without time units. However, BLM's don't get invites to cannon parties for the simple (and correct) reason of lower damage output over long periods of time due to the need to rest.
4. Accuracy vs. Precision
Data#1: 4.2 (+/- 0.1)
Data#2: 4.23567 (+/- 0.5)
This is included mainly for completeness. Accuracy is the range of values over which you can be certain that the true value of the data point can be found. Precision is the smallest increment that your device can reliably measure. More precise instruments are usually more accurate, by not definetly. Data point #1 is more accurate, and Data point #2 is more precise.
5. Error
All investigations, no matter how carefully planned, have some degree of error present. I'm not going to go into Type 1 and Type 2 errors, but suffice it to say that some errors are forgivable and others totally invalidate a study.
5a. Parametric errors: Parametric refers to a constant in your equation. This exact same error occurs every time you take a measurement or do a test. It's best if these are recognized, but it's not fatal if they are not. To explain it mathematically: X=Y, add in a parametric error and you get X+2=Y+2. Although the value of the equation has changed, the statement that they are equal (the comparison, and that's what's important) is still true. Another example would be: Your ruler is not 100% accurate and precise (none are), but since you used the same ruler every time, you can still say which piece of yarn is longer.
5b. Non-parametric errors: These errors are not the same. Your "standard" of measure or your technique changes with each test/measurement. Expressed mathematically: X=Y, add in a non-parametric error and you get X+2=Y+3. If the first equation is correct, then the second is incorrect, X+2=!Y+3.
6. Statistical Significance:
This is a huge subject. To keep it simple, let me say that just because you detect a difference between your control group and your experimental groups, it does not mean that they are actually different in any meaningful way. Remember errors and accuracy/precision? Small errors and problems with accuracy cause variations in your data collection. This means that small differences can occur even between equal groups. Allow me to throw out an arbitrary number (but one commonly used). If there is less than 5% difference between your groups, then they are essentially the same.
That's long enough for now. My next post will cover how to report findings, and will include an example of how a decent investigation could be carried out (unless I don't have time, or lose interest lol).
I don't intend to debate this part of the thread, much as I don't intend to debate if the earth is round, or if the earth revolves around the sun. This is standard operating procedure in RL.
This post came about due to a disagreement on a thread about the PCC vs. Chiv. Chain. We all had slightly different opinions on the subject. I honestly don't care which one people choose to wear, if I could afford a PCC I'd probably wear it just to look cool^^. However, it became apparent that some people are not using a valid method of parsing data. Faulty methods lead to faulty data and therefore incorrect conclusions. It's just a game, and in the grand scheme of things not too important. Nonetheless, I thought I'd try to improve things.
I could have simply posted my method said, "Trust me." Instead, I will try to persuade anyone willing to read this somewhat longish post. Do not expect a Master's Thesis on the subject. I understand that there is a wide range of age and education levels in this community, so the post will be kept somewhat simplistic.
B. The Scientific Principles
1. Skepticism: Do not accept conventional wisdom. Anything not proven by this method is only opinion. ("The Scientific Method" is the accepted norm of scientific inquiry, and has been for hundreds of years.)
2. Objectivity: Simply put, be honest and leave your ego at the door. The right answer will benefit everyone, including yourself.
3. Reproducibility: Others must be able to perform the same test and get the same results.
4. Transparency: You must disclose everything you did and how you did it, down to minute detail. This goes hand in hand with Reproducibility.
C. The Method
1. Control groups.
When you begin an investigation, you must have some hypothesis (unproven assertion, educated guess) that you want to prove. In generic terms, this can be expressed as X=!Y (=! meaning "not equal"). There must be a standard (ie, the conventional wisdom) that perhaps you think is not as good as an alternative. For this group, you will obtain data, but you will not truly experiment on this group. For the control group, you will simply observe it, and then compare the experimental group(s) to the control group.
2. Experimental Group(s)
After observing the control group, you will now begin the experimental trials. One variable, one thing and only one thing, will be changed from the control group, and you will observe and record the results. It is important that only one variable be altered at a time, otherwise you will have "confounding variables", ie. you will be observing not only the change from the control group, but also the interaction of the two variables.
In real life, and even in this game, some things are beyond our control. When this happens you end up with multiple variables, which add "error" to your data collection. This does not invalidate your study, as long as you recognize the error and can minimize or at least quantify the error. There is more on this later on.
3. Measurements
3a. Standard Units: When you record your data, you will need to do it in a format that others understand and can duplicate. In a sense, measurements are another comparison. There must be a clear, uniform and unchanging scale of measurements so that the relative values of each data point can be assessed. FF11 already provides a highly uniform scale for the effect of our actions, it is "Damage". However, this is not enough by itself.
3b. Damage/Time: That damage occurs over time, and damage without an accurate measurement of the time over which it occurs is meaningless. Consider this, you and your friend are in line at the bank to deposit paychecks. You ask him how much his check is worth, and he says $5000. Your check is only worth $3000, so you think, "That lucky SOB, heh." However, you didn't ask how long it took him to earn $5k. You get paid twice a month, but he gets paid only monthly, so in actuality you earn more money. But since you didn't ask, you don't know that and you came to the wrong conclusion. Time is another highly uniform scale of measurement, and with highly accurate digital timers readily available, everyone has the ability to measure it. Any event that occurs over a period of time must be normalized for that time (must have that time accurately measured and factored into the data). Almost all events occur over a period of time, even if that time interval is very small. Therefore, Damage/Time is the only proper method of expressing melee damage.
3c. Damage/Mob: I only mention this to refute it. Every data point has at least two coordinates (x,y). Thankfully, FF11 is not as complicated as RL, and we won't have to deal with 3 or more coordinates. For the purposes of measuring your effectiveness as a melee DDer, the x-coordinate is always damage. When you set up your method of measurements, you determine what your second coordinate will be, even if you don't realize it at the time. When you measure damage per mob killed instead of per unit of time, your y-coordinate is the mob's HP.
Allow me to explain. Events that occur over time, like melee damage, have a starting and a stopping point. You control the starting point by choosing when to engage the mob. The end point occurs when the mob dies. If you are carefully measuring time, then that end point can me expressed as a unit of time. But if your end point is only when the mob dies, untimed, then your end point is simply the exhaustion of it's HPs. If you use this method and are solo, then your Damage/Mob HP ratio is always equal to 1 (unless you lose the fight or run away). Obviously, if the value of all your data points equals 1, then you haven't collected any useful information.
If you are in in a party with other DD'ers, then the end point is (Mob's HP)-(Damage Done by Others). In this case you are competing with the other party members for damage, and you are actually comparing (measuring) your damage against theirs. Now, we all know that different mobs, even of the same type have different HP totals. Also, your fellow party members will not output damage at a constant rate; they will have damage spikes and damage troughs (lows) that will change from mob to mob. So, you are measuring damage against a non-standardized, constantly shifting "scale." Also, your test is not reproducible, since what are the odds that someone else will get all the other DD'ers in your party together again without any of them having any new merits/gear/etc? Time, however, is highly standardized and is always available for measurement.
3d. Damage per Session: This has roughly the same down-fall as Damage per mob. Your y-coordinate is (Sum of all Mob's HP)-(Sum of Other DD'ers damage), and you are probably using this method because you don't know how many mobs you fought or how long the party lasted. Again, you are measuring yourself against the shifting, non-standardized measurement of your party members.
3e. Bi-modal data points: These are events that can be expressed as simple yes/no results. These do not need to be measured over time (but can be if desired). Examples of these would be Hit vs. Miss, Critical Hit vs. Normal Hit, etc. Spell damage could (for some purposes) be measured without time units. However, BLM's don't get invites to cannon parties for the simple (and correct) reason of lower damage output over long periods of time due to the need to rest.
4. Accuracy vs. Precision
Data#1: 4.2 (+/- 0.1)
Data#2: 4.23567 (+/- 0.5)
This is included mainly for completeness. Accuracy is the range of values over which you can be certain that the true value of the data point can be found. Precision is the smallest increment that your device can reliably measure. More precise instruments are usually more accurate, by not definetly. Data point #1 is more accurate, and Data point #2 is more precise.
5. Error
All investigations, no matter how carefully planned, have some degree of error present. I'm not going to go into Type 1 and Type 2 errors, but suffice it to say that some errors are forgivable and others totally invalidate a study.
5a. Parametric errors: Parametric refers to a constant in your equation. This exact same error occurs every time you take a measurement or do a test. It's best if these are recognized, but it's not fatal if they are not. To explain it mathematically: X=Y, add in a parametric error and you get X+2=Y+2. Although the value of the equation has changed, the statement that they are equal (the comparison, and that's what's important) is still true. Another example would be: Your ruler is not 100% accurate and precise (none are), but since you used the same ruler every time, you can still say which piece of yarn is longer.
5b. Non-parametric errors: These errors are not the same. Your "standard" of measure or your technique changes with each test/measurement. Expressed mathematically: X=Y, add in a non-parametric error and you get X+2=Y+3. If the first equation is correct, then the second is incorrect, X+2=!Y+3.
6. Statistical Significance:
This is a huge subject. To keep it simple, let me say that just because you detect a difference between your control group and your experimental groups, it does not mean that they are actually different in any meaningful way. Remember errors and accuracy/precision? Small errors and problems with accuracy cause variations in your data collection. This means that small differences can occur even between equal groups. Allow me to throw out an arbitrary number (but one commonly used). If there is less than 5% difference between your groups, then they are essentially the same.
That's long enough for now. My next post will cover how to report findings, and will include an example of how a decent investigation could be carried out (unless I don't have time, or lose interest lol).
I don't intend to debate this part of the thread, much as I don't intend to debate if the earth is round, or if the earth revolves around the sun. This is standard operating procedure in RL.
Last edited:
