Gdtyjr

I have read on the several answers here and on the internet that cross-validation helps to indicate that if the model will generalize well or not and about overfitting.

But I am confused that which two accuracies/errors amoung test/training/validation should I compare to be able to see if the model is overfitting or not?

For example:

I divide my data for 70% training and 30% test.

When I get to run 10 fold cross-validation, I get 10 accuracies that I can take the average/mean of. should I call this mean as validation accuracy?

Afterward, I test the model on 30% test data and get Test Accuracy.

In this case, what will be training accuracy? and which two accuracies I compare to see if the model is overfitting or not?

This is my first question on this platform so please ignore errors.

When I get to run 10 fold cross-validation, I get 10 accuracies that I
can take the average/mean of. should I call this mean as validation
accuracy?

No. It is a [estimate of] test accuracy.

The difference between validation and test sets (and their corresponding accuracies) is that validation set is used to build/select a better model (e.g. avoid over-fitting), meaning it affects the final model. However, in your case, 10-fold CV tests an already-built model on the 10% hold-out, thus the hold-out is a test set not a validation set.

Afterward, I test the model on 30% test data and get Test Accuracy.

If you don't use the K-fold to select/build a better model, this part is not needed, run K-fold on 100% of data to get the test accuracy. Otherwise, you should keep a final test set, since the result of K-fold would be a validation accuracy.

In this case, what will be training accuracy?

From each of 10 folds you can get a test accuracy on 10% of data, and a training accuracy on 90% of data. In python, method cross_val_score only returns the test accuracies. Here is how to get both:

from sklearn import model_selection
from sklearn import datasets
from sklearn import svm

iris = datasets.load_iris()
clf = svm.SVC(kernel='linear', C=1)
scores = model_selection.cross_validate(clf, iris.data, iris.target, cv=5, return_train_score=True)
print('Train scores:')
print(scores['train_score'])
print('Test scores:')
print(scores['test_score'])

and which two accuracies I compare to see if the model is overfitting or not?

You should compare the training and test accuracies to identify over-fitting. A training accuracy subjectively far higher than test accuracy indicates over-fitting.

I suggest "Bias and Variance" and "Learning curves" parts of "Machine Learning Yearning - Andrew Ng". It presents plots and interpretations for all the cases with a clear narration.

Cross validation splits your data into K folds. Each fold contains a set of training data and test data. You are correct that you get K different error rates that you then take the mean of. These error rates come from the test set of each of your K folds. If you want to get the training error rate, you would calculate the error rate on the training part of each of these K folds and then take the average.