The role of scientific conferences in R&D

In this post I'm dealing with a very important question from the perspective of a person managing or financing R&D, how does one know how well is R&D performing? If your thought was that you'll measure it by economic success of a product that uses the results of R&D then you are on a wrong track. Namely, the product can be success or a failure because of a number of reasons, of which R&D is only one. So, another way has to be used, and actually this question is very hard. In this post I'll try to point you to a possible solution along with some of its negative sides. Before continuing, just to reiterate that this post is from the perspective of a person managing or financing R&D.

The best possible solution would be that you absolutely trust all your researchers and that they produce only the best results. But this is idealistic case, namely there are no perfect researchers, and even the best ones could produce mediocre results if they are under sufficiently high pressure. So, some form of quality assurance is necessary.

The next best solution would be for you to check what every researcher did and evaluate it by yourself, after all, whom do you trust more than yourself? But this approach also has problems, and not the small ones:

1. When good researchers does something, the only way to track him would be to do the same things he does, and that means doing his job. 
2. Even if you would know so much to be able to analyze how someone does his or her job, that wouldn't scale.
3. Finally, people tend to hate micromanagement, and this would be micromanagement.

So, this approach also wouldn't work. Another approach would be to assign for each researcher another person that would check his work. But this has almost the same problems as if you are doing everything by yourself. Especially problematic could be potential collusion between researchers, i.e. one praises other's work knowing that his own work will be reviewed, too. So, reviewers might have incentive to praise each other's work.

Thus, it is necessary to have review, but the point of the review is to be independent, done by an expert that knows the topic being reviewed and trying to be as objective as possible. You can pay independent researchers for doing review, but that's not done. What's done instead is sending papers to scientific conferences and journals where they are reviewed before being published. The review process is such that the authors don't know who reviewed their paper (blind review) or even reviewers don't know who's paper they are reviewing (double blind review). Before being published in a journal or on a conference, papers have to pass review process and authors are notified about the decision along with receiving reviewers' comments.

So, there is a way you can receive feedback about the work done by your researchers by sending them to conferences or requiring them to publish in journals. But there are additional benefits as well:

1. Even if your researches have the best intention of producing top class results, it is good to have a feedback. In the reviews there could be suggestions on how to improve the work.
2. By participating on conferences your researchers build their professional network from people doing the same or similar things and that might be very helpful on the long run.
3. You should not forget marketing aspects of scientific publications. Namely, this makes you and your people known as an organization that does research and supports their researchers which might attract new researchers and employees.

Many companies having serious R&D do publish on scientific conferences and in journals and they put on their Web pages lists of published works, here are some:

• Hitachi R&D publications
• Google Research Publications
• Microsoft Research Publications
• Facebook Research Publications
• IBM Research Publications (Haifa Lab)
• HP Enterprise Research Publications
• Sony Research Publications (Paris Laboratory)

There are many others, and I might add more to the list later.

One very important thing before I continue. People tend to think that I say that publications are mean and a goal and thus are opposing to the idea of publishing on a scientific conferences. But that's not true. Publications are only a side-product of a work who's goal is to produce something new that could be used to improve company's products!

But, nothing is perfect and so this approach has some issues you have to be aware of:

1. There are a huge number of conferences in the world many of which are at best average. You should strive to go to the best ones because there you'll receive the best feedback and also meet people that are more likely to be researching things that interest you. Which conferences are those depends on the specific research area and you have to search for them, but as a general rule of thumb the lower acceptance ratio, the better conference.
2. As I've said, the papers are only a side-product of the actual work done. But, if too great emphasize is put on conference/journal publication, then researchers start to optimize that criteria instead of doing a good work.
3. You should be careful what you publish in the papers. The moment its published, effectively it's a public knowledge. This is very good from the society perspective, but it might not be so good from the perspective of a company.
4. Publication on the conference is not so cheap. You have to pay conference fee, travel and accommodation expenses, and maybe few more things. This builds up very quickly.
5. Publication in a journal might cost nothing, but it can take time, up to 18 months. The review process for conferences is several months at most.

But in any case, I think that companies should publish as much as possible on a good conferences or in good journals as it has more benefits than drawbacks.

What is R&D according to OECD

In my previous post I wrote about my personal opinion what is R&D. In this post I'm going to analyze definition given by OECD, which might be argued to be a relevant authority for such topics. OECD produces for decades a document called Frascati Manual which is about collecting and reporting data about R&D. The latest version is from 2015 and that one is used as the basis for this post. The manual, in Chapter 2, describes what R&D is. Basically they say that the properties of R&D activity are (paragraph 2.7):

1. novel,
2. creative,
3. uncertain,
4. systematic, and
5. transferable and/or reproducible.

and activity has to satisfy all those properties to be regarded as R&D activity.

Property of the novelty can be correlated with properties 1 and 2 given in the post with my opinion. The following citations are interesting or important from the manual:

1. In the Business enterprise sector, the potential  novelty of R&D projects has to be assessed by comparison with the existing stock of knowledge in the industry. [paragraph 2.15]
2. The R&D activity within the project must result in findings that are new to the business and not already in use in the industry. [paragraph 2.15]

Those two citations mean that if you do something that anyone already does, or that anyone can do in a relatively short period of time, than it's not a product of R&D activity.

The property of creativity, i.e. the results of activities are based on original, not obvious, concepts and hypotheses can be correlated with property 2 given in the post with my opinion. The following excerpt is interesting:

An R&D project requires the contribution of a researcher!

This means that whoever is doing R&D has to have trained researches  in stuff.

The property of uncertainty, i.e. it is uncertain about the final outcome, has a direct relation to the property 5 in the post. The difference is that OECD publication claims that there are multiple dimensions to this property:

For R&D in general, there is uncertainty about the costs, or time, needed to achieve the expected  results, as well as about whether its objectives can be achieved to any degree at all. [paragraph 2.18].

Furthermore, there is discrimination criteria between R&D and non-R&D activities:

Uncertainty is a key criterion when making a distinction between R&D prototyping (models used to test technical concepts and technologies with a high risk of failure, in terms of applicability) and non-R&D prototyping (preproduction units used to obtain technical or legal certifications). [paragraph 2.18]

So, the more certain you are that there will be some functionality in the final product, less it is R&D activity!

The systematic property of R&D, i.e. to be planned and budgeted, correlates with property 4 I gave in the previous post. This, also includes keeping records, not only planning.

The final property, i.e. to lead to results that could be possibly reproduced (transferable and/or reproducible) is most interesting and I didn't include it in the elaboration of my opinion. Namely, this requires that the results be published somewhere so that conclusions can be independently verified. Somehow, it seems to me that this is the least frequent property. If nothing else, because the scientific output of companies is very small. Someone can claim here that they are publishing somewhere else, why only scientific output? The point is that under the expression scientific output I"m referring on the way the results are published, not where they are published. In other words, scientific publication includes all the necessary information in order for someone else to test the results.

For the end, just let mi note that there is another important subdivision of R&D according to OECD publication (paragraph 2.9):

1. basic research,
2. applied research, and
3. experimental development.

I'll write about those in some future post.

Using astrology to protect from APTs

Probably when you saw the title, your reaction was WTF?! Using astrology for APT detection, that's totally crazy! But, the sad fact is that it isn't so crazy after all because large number of products that are offered on the market claim that they are protecting you from APTs in the same way astrology claims it can predict your future.

To elaborate a bit more this claim, the key question is how do you know it's true that protection works? We can rephrase this question into another one: What process did manufacturers use to prove, beyond reasonable doubt, that their products are capable of detecting APTs? Did they publish somewhere what/how they did it? Also, since nothing is perfect, its obvious that no solution will detect all the cases. In how many cases will the products detect APTs, and again, if they provide such numbers, how they came up to them? What is precision, and what is recall? Anyway, this is not published so it is something you have to go buy on trust, not on the numbers and experiments.

Even more, in astrology if things turn out to be different, then the person doing prediction changes story somehow, for example he/she didn't know some crucial information which made the prediction wrong, or they predict in such a way that no matter what happens, it will be true. In other words, you can never falsify the astrology and that is the main reason it isn't science. But the same reasoning goes for products that protect you from APTs, too. Either if they protect you or not you have no way of knowing weather that was a pure luck or in the case of detection if this was something deliberately designed into the product.

So, to conclude, I don't think that majority of products for APT protection are nothing more than application of astrology to cyber security!